nossa-y/activity-frames

Turn your workday into structured workflows agents can execute. 100% local, served over MCP.

What it solves

Computer-use agents typically start every task from scratch, requiring expensive and repetitive reasoning (screenshot, reason, act) even for tasks you have performed many times. Additionally, agents lack context regarding your recent activity, meaning they start every conversation without knowing what you have been doing on your computer.

How it works

The project records screen activity locally and compiles raw snapshots into structured "activity frames"—deterministic records of tasks. These frames are then used in two ways:

  1. Context Generation: It creates compact, agent-ready context blocks (summaries of apps used, sites visited, and durations) that can be pasted into system prompts so an agent knows your day.
  2. Workflow Automation: It identifies recurring activities and converts them into executable scripts (ordered clicks and keystrokes grounded by element names and roles). This allows an agent to replay a demonstrated task instead of re-deriving it from scratch, reducing token costs and increasing reliability.

Who it’s for

Developers building computer-use agents who want to reduce token overhead and provide their agents with a factual, local memory of user activity.

Highlights

  • Deterministic Memory: Uses a two-tier contract where measured data (sessions, durations) is strictly separated from inferred interpretations.
  • Local-First Privacy: All capture, storage, and compilation happen on the local machine; no data is uploaded to the cloud.
  • MCP Integration: Includes a Model Context Protocol (MCP) server with tools for agents to retrieve context, activity patterns, and step-by-step scripts.
  • Broad Site Support: Includes entity parsers for over 25 major platforms including GitHub, LinkedIn, Slack, and Notion.