Rizzo-AI-Academy/rizzo-flow

The open, local take on Jev: typed decisions from an LLM, without generating a single token

What it solves

Rizzo Flow provides a local, open-source alternative to TypeSafe's Jev, allowing developers to get typed decisions (like yes/no, scores, or multiple-choice options) from an LLM without generating any text tokens. This eliminates the need to parse unstructured text output and provides probabilistic outcomes on your own hardware.

How it works

It uses a fine-tuned version of the Spark-X2.5 model (available in 4B and 1.7B sizes) running on llama.cpp. Instead of generating text, it performs a single forward pass to produce a probability distribution over a set of predefined answer letters. It is compatible with the Jev HTTP interface, meaning code written for Jev can be switched to Rizzo Flow by changing the API URL.

Who it’s for

Developers who need fast, typed decisions from an LLMs on local hardware (Apple Silicon, NVIDIA, AMD, Intel GPUs, or CPU) and want to avoid the latency and privacy concerns of hosted services.

Highlights

  • Zero Generated Tokens: Returns typed decisions and probabilities directly from the model's last prompt position without text generation.
  • Local-First: Runs entirely on your own machine using llama.cpp with support for various GPU backends (CUDA, Metal, Vulkan, ROCm, SYCL).
  • Fine-Tuned for Decisions: Includes a custom LoRA fine-tune to improve the shape and accuracy of probabilities compared to the base Spark-X2.5 model.
  • High Performance: Capable of making decisions in as little as 50ms to 150ms per request.
  • Jev-Compatible API: Maintains the same HTTP interface as the closed Jev service for easy migration.
  • Built-in Playground: Includes a web-based interface for testing states and questions in real-time.

Related

  • Project
    dimastatz/whisper-flow

    Whisper Flow is a FastAPI‑based service that turns OpenAI’s Whisper model into a real‑time streaming speech‑to‑text engine. Clients stream 16 kHz mono PCM audio over a WebSocket; the server buffers audio into tumbling windows, runs Whisper, and pushes back partial and final JSON transcripts with sub‑second latency (≈275 ms) and ~7 % WER on an M1 Mac. It can be run locally, via Docker, or installed as a Python package for integration.

  • Project
    Yinsongxu/LLM2Jev

    A framework that converts local LLMs and VLMs into structured decision models by extracting probabilities from logits during prefill, bypassing token-by-token decoding for lower latency.

  • Project
    mode-io/vllm-jev

    vLLM Jev is a fast HTTP serving wrapper for Jev‑style decision models (choice, yes/no, score). It runs on Linux GPUs via native vLLM and on Apple Silicon via MLX/PyTorch, handling text, images, and short videos. Install with `uv pip install .`, start a server with `vllm-jev serve <model-id>`, and query `/v1/systemone` for structured answers. Benchmarks show 4‑43× latency reductions versus reference implementations.

  • Project
    OpenDCAI/DataFlow

    DataFlow is an open‑source Python framework that lets you build, run, and share low‑code pipelines for generating, cleaning, and evaluating LLM training data. It provides a library of reusable operators, a visual WebUI, an AI‑assistant for auto‑creating pipelines, and a Ray‑based distributed execution layer.

  • Project
    ekzhang/openjev-sglang

    A server implementing the TypeSafe/Jev HTTP API using Qwen3.6-35B-A3B on SGLang, optimized for fast one-token classification and scoring.