gavamedia/deltafin

Run full Kimi K3 on a single device. And an OpenAI-compatible API server for local chat and coding agents.

Deltafin – Run the full 2.8 T‑parameter Kimi K3 model on a consumer machine

What it is – Deltafin is a single‑binary Rust runtime that lets you load and generate text with the unpruned 2.8‑trillion‑parameter Kimi K3 Mixture‑of‑Experts model (the exact weights released by Moonshot AI). The binary does all routing, tokenisation, caching and inference itself; no Python layer or external server is required. Optional add‑ons (the small Qwen model and the DSpark checkpoint) can be installed to speed up certain workloads, but the core guarantee is that every token is finally produced by K3 itself, never by a compressed or approximated version.

Key capabilities

Feature Details
Full‑model quality All 16 MoE experts are kept; no weight quantisation or pruning.
Native single binary Built with cargo; the runtime, tokeniser and router are compiled into one executable.
Cross‑platform macOS arm64 (Metal/MPS) and Linux x86‑64/aarch64 (CUDA or CPU).
Streaming & OpenAI‑compatible API deltafin serve implements /v1/chat/completions, /v1/completions and /v1/models with SSE streaming.
On‑demand model streaming setup --stream downloads only the experts that are needed, starting from ~215 GB instead of the full 1.7 TB.
Optional Qwen add‑on Small draft model (0.6 B/1.7 B) generates a guess; K3 verifies it, giving up to ~2.7× faster raw completions while keeping identical output.
Performance tracking Built‑in --stats flag prints per‑token throughput; benchmark results for Apple M1 Max are documented (≈0.29 token / s).
Health‑check utilities Verify the runtime, model files and caches after install or upgrade.
MIT‑licensed core The code you compile is MIT; model weights retain their original licences.

Quick start (from the README)

# 1. clone and build
git clone https://github.com/gavamedia/deltafin.git && cd deltafin
cargo build --locked --release

# 2. download the model (full or streamed)
# Full 1.7 TB download (fastest runtime)
./target/release/deltafin setup --full
# or stream only needed parts (starts at ~215 GB)
./target/release/deltafin setup --stream

# 3. optional Qwen add‑on for faster raw completions
./target/release/deltafin setup-qwen

Running inference

Chat (uses K3’s chat template and stops at the model’s end‑of‑turn token):

./target/release/deltafin run --chat --prompt "What are the three largest moons of Saturn?"

Raw continuation (specify a token limit):

./target/release/deltafin run --prompt "The capital of France is" --max-new 17

Add --stats to see per‑token timing.

Serving an OpenAI‑compatible endpoint

./target/release/deltafin serve --host 127.0.0.1 --port 8000

Then a client can call:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deltafin-kimi-k3","stream":true,"messages":[{"role":"user","content":"Hello!"}]}'

The server enforces a strict subset of the OpenAI API (text‑only, greedy generation, one request at a time) and returns clear error messages for unsupported fields.

Updating

./target/release/deltafin upgrade   # rebuilds the binary, keeps model files

The script checks that the working tree is clean before pulling new commits.

Documentation included in the repo

  • NATIVE‑RUNTIME.md – design of the one‑binary runtime, routing, expert pre‑fetch, tokenisation.
  • SERVER.md – exact OpenAI API fields accepted, streaming behaviour, limits.
  • PERFORMANCE.md – how the benchmark numbers (e.g., 0.29 token / s on M1 Max) were measured.
  • PLATFORMS.md – supported OS/CPU/GPU combos and required libraries.
  • HEALTH‑CHECKS.md, STORAGE.md, CONFIGURATION.md, etc.

Who built it?

Deltafin is an independent open‑source effort that layers a native Rust runtime on top of the official Kimi K3 weights released by Moonshot AI. It also incorporates community contributions such as the DSpark checkpoint (Inferact), the GigaToken tokeniser, and various SIMD/CUDA optimisations contributed by external developers.

License

The Deltafin codebase is released under the MIT license. Model weights, DSpark, and optional Qwen checkpoints retain their original licences (as listed in docs/THIRD_PARTY_NOTICES.md).

Bottom line – Deltafin is a research‑oriented tool for anyone who wants to experiment with the full Kimi K3 MoE model on a high‑end laptop or workstation, without sacrificing any of the model’s parameters for speed. It provides a compiled binary, a simple CLI, and an OpenAI‑compatible server, all documented and openly licensed.

Related

  • Dispatch
    vLLM Kimi K3 Support Preview

    vLLM is preparing day-0 open-source serving support for Moonshot AI's Kimi K3, a 2.8-trillion-parameter model featuring a hybrid KDA/full-attention architecture and native vision support.

  • Dispatch
    Qwen 3.5‑397B‑A17B release: hybrid linear‑attention MoE model with 1 M token context and state‑of‑the‑art multimodal performance

    Qwen 3.5‑397B‑A17B is a 397 billion‑parameter multimodal model that activates only 17 billion parameters per token, delivering state‑of‑the‑art performance on language, coding, reasoning and vision tasks while being up to 19× faster than its predecessor.

  • Dispatch
    vLLM Kimi K3 Support

    vLLM has released day-0 support for Kimi K3, a 2.8-trillion-parameter multimodal MoE model, featuring optimizations for Kimi Delta Attention and DSpark speculative decoding to achieve up to 370 tok/s.

  • Dispatch
    Qwen3.8-Max Release: 2.4T‑Parameter Model, Open Weights Next Week, and Broad Autonomous Capabilities

    Qwen3.8‑Max, a 2.4 trillion‑parameter model with 95 B active parameters, is now available via QwenCloud and will have its weights open‑sourced next week, delivering strong gains in coding, real‑world work, long‑horizon tasks, and multimodal agents.

  • Project
    lc2panda/alphastream

    Alphastream is an AI‑driven financial analysis web app. It uses DeepSeek/any OpenAI‑compatible LLM with LangGraph to run 14 specialized agents (technical, fundamental, sentiment, risk, etc.) plus four investor‑persona agents. Real‑time market data comes from dual adapters (AKShare & BaoStock) covering Chinese, US, Hong‑Kong, crypto and macro sources. The Next.js + React front‑end shows a dark glass‑morphism UI with chat, dashboards and chart cards, while the Flask backend streams agent progress via SSE. All services can be launched with a single Docker‑Compose command.