jmerelnyc/Photo-agents
Autonomous self-evolving agents. Vision-grounded layered memory and self-written skills for LLM agents that operate your computer.
What it solves
Photo Agents provides a framework for autonomous agents that can perceive and interact with a computer screen just as a human would. It solves the problem of LLM agents relying solely on text transcripts by grounding their actions in actual visual observations of the screen, allowing them to operate software and navigate interfaces autonomously.
How it works
The system uses a "perceive $\rightarrow$ reason $\rightarrow$ act" cycle driven by a tool-calling LLM. It processes visual input and stores observations in a layered memory system (working, global, SOP, and session archives). The agent can write its own "skills" based on successful outcomes to evolve over time. It includes a physical-execution toolset for file I/O, sandboxed code execution (Python, PowerShell, bash), and browser automation via a Chrome DevTools Protocol bridge.
Who it’s for
It is designed for users who want to deploy local, autonomous agents capable of managing their desktop environment, automating browser tasks, and executing code across different operating systems (Windows, macOS, and Linux).
Highlights
- Multi-Provider LLM Support: Native integration with Anthropic Claude and OpenAI GPT, including a failover session router.
- Self-Evolving Capabilities: Agents can create their own skills and SOPs from real-world success.
- Versatile Interfaces: Ships with multiple clients including a Streamlit web app, PyQt desktop app, and bots for Telegram, QQ, Feishu, WeCom, and DingTalk.
- Local Data Ownership: The runtime runs locally to ensure the user maintains ownership of their screen data and keys.