FluidInference/FluidUse

FluidUse – On‑device decision models for macOS

What it does
FluidUse is a Swift library that lets macOS apps automatically fill forms and make typed decisions using tiny on‑device machine‑learning models. It reads UI elements through the Accessibility API, asks a Core ML model what value belongs in each field, and types the answer back – all locally, with sub‑millisecond latency on Apple Silicon.

Key components

Component	Purpose	Model size / type
AccessibilityFormDriver / WebFormDriver	Capture a snapshot of a native app or a WKWebView form and expose its fields.	–
CuaS1FormsManager	First‑party form‑filling model (706 K parameters) converted to Core ML.	~0.7 M params
LayaManager	Decision model (322 M mmBERT) that answers choice/score/yes‑no questions in one pass, no token generation.	322 M params, multiple token‑length buckets
GLiNER2Manager	Small (128‑token) classification head for up to 8 labels.	291 MB (base) / 385 MB (multilingual)
KevFastManager	Runs the 0.8 B‑parameter Kev model (Qwen 3.5 backbone) for multi‑question answering.	~3.5 GB
ClefVisionManager	Vision‑language decision model (0.92 B) that evaluates images with typed questions.	~1.8 GB total
InternDecisionManager	Intern‑Decision 0.8 B model for typed decisions on text only.	~3.4 GB
Decision2Manager	vLLM Semantic Router “Decision 2.0” models (Kai 0.6 B, Eos 0.8 B).	1.1‑1.4 GB
Vela2Manager	Routing, safety and span‑extraction model (0.3 B).	~640 MB
ShortReplyManager	Drafts a single short social‑media reply (0.6 B).	724 MB

All managers download a checked‑in Core ML package from Hugging Face on first use, verify SHA‑256 checksums, and cache it under ~/Library/Application Support/FluidUse/Models/….

How you use it

import FluidUse
let model = try await CuaS1FormsManager.load()
let driver = AccessibilityFormDriver(application: safari)
let page = try await driver.snapshot()
for field in page.elements where field.isActionable {
    let ctx = FormSchema.renderContext(formTitle: page.title, element: field)
    let decision = try await model.score(context: ctx, options: …)
    // decode → driver.type / driver.click
}

The same pattern works for the other managers (Laya, GLiNER2, Kev, etc.) – you load the manager, call answer/score/classify, then act on the result.

Performance highlights

• Form‑filling model: ~0.9 ms per decision on the Neural Engine, accuracy identical to the PyTorch reference.
• Laya: 3.6 ms per short question on an M5 Pro (≈7× faster than GPU PyTorch).
• GLiNER2 classification: 1.6 ms per two‑option decision (FP16) on the Neural Engine.
• Kev 0.8 B: 18 ms for a two‑question ticket on M5 Pro.
• Clef‑Vision: 260 ms for one image + ~500 tokens (vision 50 ms, language 190 ms).
• Decision 2.0 (Kai): 17 ms for a ~1 k‑token request; Eos a bit slower.
• Short‑reply draft: ~190 ms per reply on M5 Pro.

All timings are measured on Apple Silicon (M5 Pro) with on‑device Core ML execution; nothing is sent to a server.

Demo & tooling

• FluidUseDemo – CLI that picks a running app, loads a profile and fills the form.
• Several Swift‑UI demos (LayaTetrisDemo, GLiClass2048Demo, GuardrailDemo, FrontDoorDemo, ShortReplyDemo).
• CLI helpers for benchmarking, running the models on synthetic suites, and playing simple games (Tetris, 2048) to showcase decision latency.

Who it’s for
Developers building macOS automation, on‑device assistants, or privacy‑first UI‑automation tools that need ultra‑fast, locally‑run decision models.

License
Apache 2.0 for the FluidUse code; the CUA‑S1‑FORMS model is MIT‑licensed.