jmerelnyc/Photo-agents

Autonomous self-evolving agents. Vision-grounded layered memory and self-written skills for LLM agents that operate your computer.

What it solves

Photo Agents provides a framework for autonomous agents that can perceive, reason, and act on a computer screen. It aims to replace long chat transcripts with a biologically inspired memory system and vision-based grounding, allowing agents to operate a computer similarly to a human user.

How it works

The system uses a streaming agent loop that drives a tool-calling LLM through a cycle of perceiving the screen, reasoning about the state, and taking action. It features a multi-provider LLM router supporting Anthropic Claude and OpenAI GPT. To interact with the computer, it employs a physical-execution toolset including file I/O, sandboxed code execution (Python, PowerShell, bash), and browser automation via a Chrome DevTools Protocol bridge. It also implements a layered memory system (working, global, SOP, and session archive) and a self-evolving mechanism where agents can write their own skills from successful outcomes.

Who it’s for

Users who want to run autonomous, vision-grounded agents locally to automate computer tasks across various interfaces, including web apps, desktop applications, and various chat platforms (Telegram, QQ, Feishu, etc.).

Highlights

  • Vision-Grounded Action: Operates based on what it sees on screen rather than just text transcripts.
  • Multi-Frontend Support: Includes a polished Streamlit web app, PyQt desktop app, and multiple chat-bot integrations.
  • Multi-LLM Router: Native support for Claude and GPT with a failover session mixin.
  • Self-Evolving Skills: Agents can create and write their own skills and SOPs based on real success.
  • Local Execution: Runs locally to ensure ownership of screen data and keys.

Related

  • Project
    Anionex/agent-vision-toolkit

    A toolkit that gives text-only LLM agents visual capabilities through task-aware vision tools and seamless proxy integrations for image Q&A and GUI automation.

  • Project
    zai-org/GLM-V

    A series of vision-language models (VLMs) that enhance multimodal reasoning, supporting native function calling, visual grounding, and complex document understanding.

  • Project
    GetStream/Vision-Agents

    A framework for building low-latency, multi-modal AI agents that can watch, listen, and understand video in real time using a combination of CV models and LLMs.

  • Project
    lsdefine/GenericAgent

    A minimal, self-evolving autonomous agent framework that allows LLMs to control a local computer and autonomously build a personal library of reusable skills.

  • Project
    browser-use/browser-use

    A framework that enables AI agents to control a web browser like a human, allowing them to perform tasks such as filling forms and extracting data through plain-language instructions.