TheoLeeCJ/SemIf-OpenJev

What it solves

SemIf provides a way to make fast, typed decisions using open-source LLMs without the overhead of generating full text responses. It eliminates the need for the model to generate a sentence or JSON object that software then has to parse, which typically slows down agentic workflows where decisions are small (e.g., routing a request or retrying a task).

How it works

Instead of sampling tokens autoregressively, SemIf reads the probabilities (logits) of specific typed options directly from the model's output layer in a single forward pass.

Key technical features include:

• Runtime-defined criteria: The decision criteria and options are provided with the request.
• Shared-state awareness: A long context (state) can be prefetched once and then branched across multiple different decision criteria in parallel, significantly increasing throughput.
• Multi-backend support: It supports PyTorch, Apple Silicon (MLX/MPS), and CPU-only execution via llama.cpp.
• Temperature Calibration: Includes per-workload temperature scaling to ensure the model's confidence (probabilities) matches its actual accuracy.

Who it’s for

Developers building AI agents or automated workflows that require high-speed, reliable, and auditable semantic decisions based on structured options.

Highlights

• High Speed: Direct logit readout is significantly faster (up to 5.21Ñ faster) than generating a compact JSON array of answers.
• Open-Source Alternative: Reproduces the interface pattern of closed services like Jev.
• Efficient State Reuse: Supports serial prefix reuse and parallel suffixes to handle multiple decisions over the same state efficiently.
• Browser-Ready: Includes a WebGPU demo for immediate testing without a waitlist.