valentinfrlch/ha-llmvision

LLM Vision – Visual intelligence for Home Assistant

What it is – A Home Assistant integration that lets your smart‑home “see”. It sends snapshots, video clips or live camera feeds to a multimodal large‑language model (LLM) and returns natural‑language descriptions, answers, or summaries. The results are stored as sensors and can be displayed on dashboards or queried by Home Assistant’s Assist.

Key capabilities

• Multimodal LLM support – works with OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure, Groq, Ollama, Open WebUI, LocalAI and any service exposing an OpenAI‑compatible API.
• Image / video analysis – describe a single picture, a video file, a live camera stream, or a Frigate event based on a custom prompt.
• Memory – can remember identified people, pets and objects across events.
• Timeline – stores each analyzed event in a chronological list that can be shown with a Timeline Card on the Home Assistant UI.
• Sensor updates – automatically creates/updates entities (e.g., sensor.living_room_last_seen) with extracted information.

Quick start (HACS)

1. Add the integration via the Home Assistant Community Store (HACS) – it’s in the default repository.
2. Restart Home Assistant.
3. In Settings → Devices & Services search for LLM Vision and add it.
4. Point the integration to Home Assistant’s /media folder (used for storing temporary snapshots). If you run Home Assistant in a container, mount a host folder to /media.
5. Open the integration page, click Add Entry, and configure your LLM provider (API key, endpoint, model, optional system prompt).
6. (Optional) Install the provided Blueprint to get AI‑summarized camera‑event notifications and a ready‑made Timeline Card.

Typical workflow

• A motion event triggers a camera snapshot (or Frigate records a clip).
• LLM Vision uploads the media to the configured LLM, adds your prompt (e.g., “Describe what’s happening and name any people or pets”), and receives a text response.
• The response is saved to a sensor and, if enabled, appended to the timeline.
• You can query the information via Home Assistant’s voice assistant: “What did the front‑door camera see at 3 p.m.?”

Resources

• Website & docs – https://llmvision.org and the GitBook guide https://llm-vision.gitbook.io/getting-started
• Examples / gallery – https://llmvision.org/gallery/
• Community – Home Assistant forum thread and Discord server (links in the README)
• Support – star the repo or buy the author a coffee via the BuyMeACoffee badge.

Who it’s for – Home Assistant users who want to add visual AI capabilities (security cameras, pet monitoring, object detection) without writing code, and who already have access to an LLM API or a local model.