Flide 0.1.10 release notes: High-speed inference for Jev-style decision models

Flide is an inference infrastructure specifically optimized for Jev-style decision models. By implementing a technique called "exact state forking," the engine significantly reduces latency, achieving up to 9.5x faster inference speeds compared to standard engines like vLLM and SGLang while maintaining output exactness.

Core Technology: Exact State Forking

The primary performance gain in Flide comes from its approach to handling multiple questions derived from a single input state. In traditional LLM inference engines, repetitive processing of shared input data creates a bottleneck. Flide optimizes this by computing the shared input only once and then forking the state to answer multiple questions in a single forward pass. This architecture is particularly effective for decision-making tasks where a single context (e.g., a customer support ticket or a data log) requires multiple distinct classifications or actions.

Performance and Compatibility

Flide is designed to serve the Jev /v1/systemone API, making it a drop-in replacement for existing Jev-style workflows. The engine supports a wide range of decision models, including:

  • Model Families: Support for models such as OneJev, Qwen3.5, Gemma 4, and various specialized decision models like Ainvest-Decision, KnowLine, and Perplexity's decider.
  • Hardware Efficiency: The engine is built to maximize throughput on modern GPUs, with specific optimizations for FP8 quantization on Ada, Hopper, and Blackwell architectures.
  • Benchmarking: Performance gains vary by model and workload. For instance, while some models see modest improvements, others—such as those handling complex MultiRC or QuALITY tasks—demonstrate speedups exceeding 9x compared to baseline engines.

Getting Started

Flide is available via PyPI and can be installed using uv or pip. The engine provides a CLI for serving models, which simplifies the deployment process for local development or production environments.

uv venv -p 3.12 && source .venv/bin/activate
uv pip install flide torchvision
flide serve --model OmniJev/OneJev1.6-0.8B

Once the server is running, the /v1/systemone endpoint accepts JSON payloads containing a shared state and multiple questions, allowing the engine to execute its optimized forking logic.

Configuration Options

Flide includes several flags to tune performance for specific hardware and use cases:

  • --session-cache-gb: Caches previous request states to minimize re-computation for sequential agent steps.
  • --batch-invariant: Ensures deterministic output regardless of batching, trading off some speed for consistency.
  • --frontends: Offloads HTTP, rendering, and tokenization to separate processes to handle higher client concurrency.

Community Insights

Developers exploring Flide have noted its utility for datasets where a single input triggers multiple decision-based queries. The maintainers emphasize that even for single-question scenarios, Flide can achieve up to 3x speed improvements depending on input length and model architecture. For those interested in the underlying research, the team has published their methodology in the paper "Flide: Accelerating Decision Inference with Exact State Forking" (arXiv:2610.42148).

Sources