Ephemeral-AI-Lab/layerfs

LayerFS – A SQLite‑backed, content‑addressed “time‑machine” for AI agent workspaces

What it is

• A library, CLI, and FUSE‑based filesystem that lets an AI agent (or any tool that touches the file system) work inside an ephemeral workspace.
• When the agent finishes a step, the workspace can be committed – turning the whole filesystem snapshot into an immutable Commit.
• Commits are stored in a LayerStack: a history of layers built on top of each other using three classic techniques:
  1. Content‑addressed storage (CAS) – objects are named by the hash of their bytes, so identical data is stored only once.
  2. Content‑defined chunking (CDC) – files are split into stable chunks; a tiny edit only creates a new chunk, not a whole new copy of the file.
  3. Copy‑on‑write (COW) – only the changed directories and files are rewritten when a new layer is published.
• The whole history lives in a single SQLite database (the Store). The database is read‑only for agents; only the LayerFS runtime writes to it when a commit is published.

Why it matters for AI

• Modern autonomous agents (LLM‑driven planners, reinforcement‑learning‑from‑human‑feedback loops, multi‑agent simulations) need to try many what‑if variations of code, data, or configuration.
• Re‑creating a full copy of a large workspace for each trial is wasteful. LayerFS lets agents fork a previous layer instantly (zero‑copy) and run many parallel roll‑outs, then promote the best branch.
• Because the history is immutable and content‑addressed, the system can deduplicate massive amounts of data across thousands of trials, dramatically reducing storage and I/O.

Key components (as of the v0.1.6 developer preview)

Component	Role
Storage engine	Manages CAS, CDC, COW, object verification, and the SQLite Store.
SDK	Rust library exposing types like Client, LayerStack, Branch, Workspace, and functions to create/commit/query them.
CLI (layerfs-cli)	Command‑line interface for creating a Store, initializing a LayerStack, spawning workspaces, committing, and inspecting history.
Filesystem projection	Provides a mount point (via host materialization or a FUSE container) that appears as a normal directory to the agent.
Container runtime	Spins up lightweight Linux containers with FUSE, keeping the mutable workspace inside the container while the durable Store stays on the host.

Typical workflow

1. Create a Store – layerfs db create store.sqlite.
2. Initialize a LayerStack – layerfs layerstack init --name demo --empty creates a genesis layer.
3. Start a Workspace – layerfs workspace exec <branch> launches a process (or container) with a private view of the filesystem.
4. Do work – the agent runs its tool, writes files, etc.
5. Commit – layerfs commit <workspace> turns the mutable state into an immutable Commit and attaches it to the current Branch.
6. Branch / fork – create new branches from any previous Commit to explore alternatives without copying data.

Performance highlights (v0.1.6)

• Adding 10 bytes to a 32 MiB file only increased the logical store size by ≈ 0.026 MiB (thanks to CDC).
• Six deterministic edits across a workspace grew the logical store by ≈ 0.225 MiB.
• All measurements come from a real‑FUSE benchmark suite; the underlying SQLite pages may grow more, but logical content stays tiny.

Limitations (important to know)

• Developer preview only – no crash‑ or power‑loss durability guarantees; keep backups of important data.
• One Store per client; there is no cross‑host synchronization yet.
• Commits are not automatically fsynced; the sandbox’s mutable state lives only in memory until a commit is published.
• Only a single construction worker builds the immutable objects, making large batch imports slower than earlier versions.
• No explicit compaction; old stores remain readable but the system does not reclaim space automatically.
• CLI JSON output is a provisional format, not a stable API.

Getting started

# Clone the v0.1.6 tag (the preview code lives under `crates/`)
git clone https://github.com/Ephemeral-AI-Lab/layerfs.git
cd layerfs && git checkout v0.1.6

# Build the CLI (requires Rust 1.85+, macOS/Linux)
cargo build --release -p layerfs-cli
export LAYERFS_BIN=$PWD/target/release/layerfs
export LAYERFS_CONTEXT=$PWD/.layerfs/context
mkdir -p $PWD/.layerfs

# Create the SQLite store and a demo LayerStack
$LAYERFS_BIN db create $PWD/.layerfs/store.sqlite
$LAYERFS_BIN context use --store $PWD/.layerfs/store.sqlite
$LAYERFS_BIN layerstack init --name demo --empty
$LAYERFS_BIN query layerstacks

Continue with the version‑specific quickstart guide for branch creation, workspace execution, and optional FUSE container usage.

Roadmap snapshot

• 0.2.0 (planned) – portable projection layer (reflink/clonefile, future OverlayFS support).
• Later – cross‑host store export/import, synchronization, and richer runtime integrations.

Community & support

• Discord: https://discord.gg/DrgJ4DX9E
• WeChat group (QR code in the repo) – invitation expires periodically.
• Follow the project lead on X: @yifanxu_ephai

License – MIT.

---

LayerFS is a research‑grade tool aimed at developers building autonomous AI agents that need cheap, versioned filesystem state. It is not a production‑grade storage system yet, but it provides a concrete platform for experimenting with “ephemeral workspaces + durable shared history” patterns.