HKUDS/CatchMe

CatchMe – Personal Memory Engine for AI Agents

What it does

Runs in the background on your computer and records everything you do: mouse clicks, keystrokes, window focus changes, clipboard contents, screenshots, notifications, and file modifications.
The raw events are automatically organized into a hierarchical activity tree (Day → Session → App → Location → Action).
Small LLM‑driven summarizations are added at each level, turning the tree into a searchable knowledge base.
When you ask a question (via CLI, a web dashboard, or an AI‑agent skill), the LLM walks the tree top‑down, picks the most relevant branches, inspects the underlying raw data, and returns a concise answer.
All data stays local (SQLite + FTS5) and can be processed with any LLM you like – from cloud APIs (OpenAI, Anthropic, Gemini, etc.) to offline models served by Ollama, vLLM, or LM Studio.

Why it matters

Gives AI assistants (Claude, Cursor, OpenClaw, NanoBot, …) a personal memory of your digital life without needing vector databases or external services.
Keeps privacy intact: nothing is uploaded unless you deliberately use a cloud LLM for summarization.
Works on macOS, Windows, and Linux (X11 only) with a tiny runtime footprint (~0.2 GB RAM).

Key Features (as described in the README)

Feature	What you get
Always‑On Event Capture	Six low‑level recorders capture mouse, keyboard, window focus, clipboard, notifications, and file changes instantly (no timers).
Intelligent Memory Hierarchy	Events are auto‑organized into a five‑tier tree; each node gets an LLM‑generated summary for fast, semantic browsing.
Tree‑Based Retrieval (no vectors)	Instead of embedding vectors, the LLM directly navigates the activity tree, selecting branches and drilling down to raw evidence.
Zero‑Config Agent Integration	Drop a single skill file into any CLI‑based AI agent; the agent can query memories via simple CLI commands.
Ultralight & Privacy‑First	Stores everything locally in SQLite + FTS5; can run fully offline with local LLMs.
Rich Web Interface	Interactive dashboard with timelines, tree navigation, and a chat window to converse with your own digital footprint.
Cost‑Control Settings	Configurable limits on LLM calls, image count per cluster, and summarization frequency to keep token usage low.

How It Works (high‑level pipeline)

Capture – Six background daemons listen to OS events (mouse, keyboard, window focus, screenshots, clipboard, notifications).
Index – Events are streamed into a Hierarchical Activity Tree.
Summarize – A configurable LLM creates short summaries for each node (Day, Session, App, Location, Action).
Retrieve – When you ask a question, the LLM reads the top‑level summaries, selects the most promising branches, and recursively descends until it finds concrete evidence (e.g., a screenshot or keystroke log) to answer.

Getting Started (quick‑start steps from the README)

# 1. Clone & create a Python 3.11 env
git clone https://github.com/HKUDS/catchme.git && cd catchme
conda create -n catchme python=3.11 -y && conda activate catchme

# 2. Install the package (editable mode)
pip install -e .

# 3. Platform‑specific permissions
#   macOS – grant Accessibility, Input Monitoring, Screen Recording
#   Windows – run the terminal as Administrator
#   Linux – install xdotool & xprop, then `pip install -e "[linux]"`

# 4. Initialise LLM configuration (interactive)
catchme init   # choose provider, API key, model, etc.

# 5. Start recording
catchme awake   # daemon runs in the background

# 6. Query or explore
catchme ask -- "What was I coding yesterday?"
catchme web    # opens the local dashboard at http://127.0.0.1:8765

LLM Configuration (what you need to set)

Provider – any of the many supported services (OpenAI, Anthropic, Gemini, Ollama, etc.).
Model – name of the model you want to use for summarization and retrieval.
Context window – must be large enough for the max_tokens_* settings (default up to 8192 tokens for final answers).
Privacy note – if you use a cloud provider, raw activity data will be sent for summarization; otherwise keep everything offline with a local model.
Cost limits – llm.max_calls, filter.mouse_cluster_gap, and other knobs let you cap token usage.

Agent Integration

CatchMe can be exposed to any CLI‑based AI agent via a tiny skill file:

Light skill – you run catchme awake yourself; the agent only needs to call catchme ask ….
Full skill – the agent can start/stop CatchMe and use the MCP stdio server for richer interaction.

Supported tools for agents:

search_activity(query, date?)
list_days()
get_session(session_id)
get_tree(date)

Community & Resources

Blog / design deep‑dive – https://hkuds.github.io/CatchMe/
Discord – https://discord.gg/2vDYc2w5
Feishu / WeChat groups – links in COMMUNICATION.md
License – Apache 2.0

Bottom line

CatchMe is a lightweight, privacy‑first “personal memory store” that turns your everyday computer activity into a structured, LLM‑queryable knowledge base. It lets you give any AI assistant a genuine sense of your digital history without the overhead of vector databases or cloud storage.