Roy3838/Observer

What it solves

Observer AI allows users to create "micro-agents" that passively monitor a computer's environment—including the screen, camera, and audio—and automatically trigger actions or notifications based on what they observe. It eliminates the need for manual monitoring by automating the process of logging, reacting, and alerting.

How it works

Users build agents by combining three components:

1. Sensors: Inputs that capture data, such as screenshots, camera feeds, OCR (text recognition) from the screen or camera, clipboard content, and audio transcriptions (via Whisper).
2. Models: Local LLMs (via llama.cpp, Ollama, or Transformers.js) that process the sensor data based on a system prompt.
3. Tools: JavaScript functions that the agent can execute based on the model's output. These include sending notifications (Email, Discord, Telegram, WhatsApp, SMS), managing agent memory, recording video clips, and interacting with the system (mouse clicks, browser notifications).

Who it’s for

It is designed for users who want to automate environmental monitoring and notification workflows while keeping their data private by using local inference.

Highlights

• Multimodal Sensing: Supports screen, camera, and audio inputs for comprehensive observation.
• Local Inference: Compatible with Ollama, llama.cpp, and Transformers.js to ensure data privacy.
• Extensive Notification Suite: Integrated tools for Discord, Telegram, WhatsApp, SMS, and Email.
• Programmable Logic: Uses a JavaScript sandbox to allow custom reactions to model responses.
• Intelligent Memory: Agents can store and retrieve text and image-based memory.