mrsarac/ff-tracking

ff‑tracking – a code‑driven visual‑effects demo

What it is – A Rust‑based demo that programmatically creates a short video of a virtual camera filming a terminal‑style AI‑agent UI. The camera tracks every glyph the agent “types”, applying realistic‑looking depth‑of‑field, lens‑flare, colour‑palette swaps and sound that is generated from the same data. The whole pipeline is driven by the fframes library (a Rust video‑generation framework) and a custom SkSL shader that runs on Skia Metal (macOS) or Vulkan (Windows).

Key components

• hud crate – defines the “screen” (glyph positions, camera parameters, palette) and writes JSON files with per‑frame box coordinates and camera transforms.
• Pass 1 – flat – renders a 3840×2160 SVG‑like representation of the terminal, one glyph per SVG element, so the exact pixel location of every character is known.
• Pass 2 – lens – a SkSL shader consumes the Pass 1 frames as iChannel0 and simulates a physical camera: perspective, depth‑of‑field (36‑tap golden‑angle bokeh), colour‑shift, bloom, grain and a “tracker” overlay that draws boxes, links and a lock‑on effect.
• Audio generation – tools/sfx.py reads the tracker JSON and synthesises a minimal soundtrack (screen hum, keystroke clicks, lock beeps, cuts) with NumPy/SciPy, then normalises loudness with FFmpeg.
• Optional Slab score – score/score.py builds a richer 6‑second musical piece in the Slab DAW, synchronised to the same JSON data.
• Build scripts – tools/render.sh builds the workspace, runs the two passes, creates the audio, and assembles out/tracking.mp4. Helper commands let you preview, extract frames, or stop the shader after a given stage.

Why it matters – It showcases how a deterministic, data‑first rendering pipeline can be used to create cinematic‑style visualisations of AI‑agent “thought‑processes”. The project is a concrete example of programmatic video generation, combining computer‑vision‑style tracking, GPU shaders, and procedural audio – all driven by Rust code rather than manual key‑framing.

Getting started – On Apple‑silicon macOS install Rust, FFmpeg and Python 3, then run:

git clone https://github.com/mrsarac/ff-tracking && cd ff-tracking
tools/render.sh   # produces out/tracking.mp4

Windows users follow the extra steps in docs/WINDOWS.md (Vulkan backend).

Customization – The SCENE_LIST in hud/src/lib.rs lets you edit the typed text, colour palette, camera tilt/zoom and font size. After editing, re‑run tools/render.sh and all tracker boxes, focus shifts and audio cues update automatically.

License – MIT (except the optional Slab score which depends on GPL‑3.0 slabkit).