ztxz16/fastllm

FastLLM – High‑Performance Local LLM Inference Engine

What it is – FastLLM is a C++‑based runtime that lets you run large language models (LLMs) on a single machine or expose them as a service. It does not rely on PyTorch; instead it ships its own kernels for CUDA, ROCm, CPU and NUMA, and can even place parts of a model on SSDs. The project ships a Python package (ftllm) that provides a CLI, a web UI, and an OpenAI‑compatible API server.

Key capabilities

Feature	What it means for you
Fast model onboarding	Supports many recent Chinese‑origin models (Qwen‑4‑Exp, Qwen‑3.8‑Flash‑Next, DeepSeek‑V4, Kimi‑K3, GLM‑5, etc.) out‑of‑the‑box.
Mixture‑of‑Experts (MoE) & hybrid deployment	Ordinary layers can run on GPU while expert layers can be placed on CPU, NUMA memory or even disk, allowing very large models to run on machines with limited GPU memory.
Multi‑GPU tensor parallelism	Works with any number of GPUs, including odd counts, and can automatically detect available devices (--tp auto).
Speculative decoding	Built‑in support for MTP, DSpark and DFlash2 – techniques that generate “draft” tokens ahead of time to speed up generation.
Broad precision support	FP16, BF16, FP8, INT4, K‑Quant, AWQ, GGUF, etc., with automatic dtype selection (--dtype auto).
OpenAI‑compatible API	Exposes /v1/chat/completions, /v1/completions and Anthropic‑style endpoints; supports streaming, tool calls, and custom sampling parameters.
Extensible back‑ends	Core kernels are C++/CUDA, but you can plug in Triton ops, custom Python model graphs, or other accelerators.

Typical workflow

1. Install – pip install -U ftllm (pre‑built wheels for Linux/Windows GPU, ROCm or source build for other hardware). Windows may need an extra dependency wheel.
2. Smoke test – ftllm run Qwen/Qwen3-0.6B downloads a tiny test model and runs a single inference.
3. Start a service – ftllm server <model> launches an OpenAI‑compatible server (default 0.0.0.0:8080).
4. Optional UI – ftllm webui --api_base http://127.0.0.1:8080/v1 or simply ftllm launch which opens a browser‑based “Launcher” that also helps you download models, tune hardware settings, and manage agents.
5. Call the model – standard curl request to the /v1/chat/completions endpoint.

How to tune performance (examples from the README)

• Tensor parallelism: --tp 0,1 or --tp auto.
• Hybrid MoE: --device cuda --moe_device numa or even --moe_device "{'cuda':1,'numa':8,'disk':1}".
• Context length: --max_context_length 65536 together with a RoPE scaling JSON to safely extend the window.
• Speculative decoding: --mtp 4 (MTP draft tokens) or --draft /path/to/draft --draft_tokens 5 for external checkpoints.
• Cache control: --kv_cache_dtype fp4, --gpu_mem_ratio 0.9, --tokens 1000000 for large‑scale KV cache.

Documentation & community

• Detailed model‑specific deployment guides (Qwen, DeepSeek, Kimi, GLM, Laguna, etc.).
• Benchmark tables per model showing TTFT, prefill and decode throughput.
• A QQ/WeChat community for users and developers.
• Apache‑2.0 license, welcoming contributions via Issues and PRs.

Bottom line – FastLLM is a production‑ready, high‑throughput inference engine for large language models that runs locally without PyTorch, supports advanced features like MoE‑mix, speculative decoding, and provides a full stack (CLI, web UI, OpenAI‑compatible API) for developers and researchers who need to serve big models on commodity hardware.