Ambient Context: Local Text-Based Activity Logging for LLM Memory

Ambient Context is a macOS application designed to create a persistent, written record of a user's digital activity to serve as long-term memory for Large Language Models (LLMs). Unlike traditional activity trackers that rely on screenshots or video, Ambient Context extracts text directly from the focused window using the macOS accessibility tree, saving the data as local Markdown files.

Local-First Architecture and Privacy

Ambient Context operates on a strict local-first privacy model to ensure that sensitive user data never leaves the machine. The application does not require an account, does not use a server, and contains no bundled models or telemetry.

Key privacy protections include:

  • Focused Window Only: The app only reads the window currently in focus; it ignores background, minimized, or off-screen windows, and ceases operation when the screen is locked.
  • Automatic Redaction: Password managers and private browsing windows are excluded from capture. Secure input fields are skipped at the accessibility level, and the app scrubs credentials, API keys, and credit card numbers before writing to disk.
  • User-Owned Data: All records are stored as plain Markdown files in a user-selected folder, allowing for easy grep searches, manual deletion, or migration.

Technical Implementation via Accessibility API

Rather than using Optical Character Recognition (OCR) on screenshots, Ambient Context leverages the macOS accessibility API to read the text of the focused window every few seconds. This approach minimizes storage overhead and avoids the privacy risks associated with image-based capture.

Compatibility and Limitations

Because the app relies on the accessibility tree, its effectiveness varies by application:

  • Chromium and Electron Apps: Applications like Chrome, Slack, VS Code, and Obsidian only build their accessibility tree when requested. This can result in "thin" initial captures that fill in over several seconds. Chrome may also exhibit slight window-resize animations during capture.
  • Terminals: GPU-rendered terminals such as Kitty and Alacritty provide little to no text via this API. Standard tools like Terminal.app and iTerm2 are fully supported.
  • System Requirements: The app requires macOS 14+ on Apple Silicon.

LLM-Optimized Data Format

The captured data is structured specifically to be ingested by AI agents (such as Claude Code) to answer questions about past work or automate status reports.

Each day is recorded in a single Markdown file with the following characteristics:

  • Timeline Structure: Block headings organize the day's activity by time, application, and document title (e.g., ## 09:41–10:05 · Chrome · Tauri tray documentation).
  • Deduplication: Body lines are written only once per day regardless of how often they are seen, preventing the file from growing too large for an LLM's context window.
  • Contextual Metadata: Each block records the document path or URL, allowing an agent to reference the original source rather than relying on fragments.
  • Agent Guidance: The capture folder includes an AGENTS.md file that explains the data format to any LLM reading the directory.

Community Perspectives and Trade-offs

Discussion among developers suggests that while the accessibility API approach is lean, it faces challenges regarding data "lossiness."

"The accessibility tree works for a lot of things but the more I worked on it the more stuff I found needed pixels."

Some users noted that many applications are not "wired up right" for accessibility, which may require custom profiles for specific apps to achieve high-fidelity capture. Others suggested that integrating with structured digital footprints—such as git commits, Slack messages, and emails—might provide a more reliable alternative to screen scraping for certain workflows.

Sources

Related

  • Project
    cyberchitta/llm-context.py

    A context curation tool that enables AI coding agents to automatically identify, pack, and manage the specific files and code excerpts needed for a task to avoid token limits and missing information.

  • Project
    openTrinity/mycontext

    MyContext is a desktop app that continuously ingests a user’s work data (chat, docs, meetings, etc.) into a private, locally stored knowledge graph. The graph can be queried by LLMs, local agents, or authorized AI apps, providing them with a personal context layer for more informed responses. Features include a digital‑self assistant, natural‑language search with source citations, and strict human‑in‑the‑loop controls. The project is actively developed and released under the Elastic License 2.0.

  • Dispatch
    Anthropic Effective Context Engineering for AI Agents

    Anthropic introduces context engineering as a strategy for curating the optimal set of tokens to maximize LLM performance and overcome attention scarcity and context rot.

  • Project
    nossa-y/activity-frames

    activity‑frames records your screen locally, compiles raw snapshots into deterministic “activity frames”, and exposes them as compact context blocks or executable scripts for AI agents, dramatically reducing token cost and improving reliability of repeated computer‑use tasks.

  • Dispatch
    Managing LLM Context Windows: Avoiding the 'Dumb Zone'

    Large advertised context windows in LLMs often suffer from performance degradation, leading developers to adopt 'smart zone' management strategies like manual compaction and artifact-based handoffs.