nossa-y/activity-frames

activity‑frames – turning your screen‑time into structured memory for AI agents

What it does – activity‑frames records everything you do on your computer (window titles, URLs, clicks, keystrokes, etc.) locally, then compiles those raw snapshots into activity frames: deterministic, time‑bounded records of each task. The frames can be rendered as a concise, token‑efficient context block that you paste into a prompt, or exported as JSON/YAML for an agent to replay the exact sequence of actions.

Why it matters – Current computer‑use agents re‑derive every task from scratch each time, which burns a lot of LLM tokens and can be flaky. By turning repeated work into a reusable script, activity‑frames lets agents execute known workflows at (near) zero token cost and with perfect fidelity, while still giving them a high‑level summary of the rest of your day for context.

Key features

• Local‑only capture – a bundled screen recorder (nocta‑recorder) stores snapshots in a SQLite DB; nothing is uploaded.
• Deterministic compilation – pure code (no LLM) turns millions of rows into a few hundred tokens in < 1 s.
• Agent‑ready context blocks – ready‑to‑paste YAML/markdown summaries that fit within LLM token limits.
• Executable workflows – aframes steps --find "…" extracts an ordered click‑by‑click script that agents can replay without re‑reasoning.
• MCP integration – six Model‑Context‑Protocol tools (get_context, get_activity, get_steps, get_day_summary, get_patterns, get_communications) expose frames to any MCP‑compatible agent.
• Python API – ActivityLog() provides day(), recent(), and context() helpers.
• Privacy controls – audio off by default, text content omitted unless --include‑text is passed, and all data stays on your machine.
• Site parsers – built‑in entity extraction for 25+ popular sites (LinkedIn, GitHub, Google services, YouTube, Slack, Notion, etc.).

Installation

pip install activity-frames          # core library & CLI
pip install "activity-frames[yaml]" # optional YAML output support

The first aframes record call will download and verify the appropriate nocta‑recorder binary for macOS (Apple Silicon or Intel). Linux users can point $AFRAMES_DB at any compatible capture DB.

Typical CLI workflow

# 1. Record your screen (audio off by default)
aframes record

# 2. When you need to give an agent context about the last 2 h
aframes context --hours 2   # prints a ready‑to‑paste block

# 3. Extract a reusable workflow (e.g., sending a LinkedIn message)
aframes steps --find "message john doe"

# 4. Run the deterministic executor from an agent via MCP
aframes mcp   # serves the six MCP tools over stdio

Python usage example

from activity_frames import ActivityLog
log = ActivityLog()
print(log.context(hours=3))          # one‑liner for a prompt
print(log.recent(hours=1).json())    # raw frames as JSON

Paper & community – The approach is described in Activity Frames: Deterministic Screen‑Activity Compilation for Agent Memory and Replay (arXiv 2608.05784). The repo includes the measurement code, a replay executor, and extensive docs (AGENTS.md, SPEC.md, docs/*). It is MIT‑licensed and actively maintained (v0.2, macOS‑first, Linux support via custom recorder).