TheoLeeCJ/SemIf

What it solves

SemIf provides a faster, more efficient way for AI agents to make small, typed decisions (such as routing a request or retrying an operation) without the overhead of generating full text responses. Instead of asking a model to generate a "yes" or "no" and then parsing that text, SemIf reads the probabilities of specific options directly from the model's internal state (logits), eliminating the need for a decoding loop or JSON repair.

How it works

SemIf uses a "decision-native" approach where criteria and options are provided to a model in a single request. The system reads the native option logits to determine probabilities for the provided options, rather than sampling tokens. It also supports "shared-state awareness," allowing a single long context (state) to be prefetched once and then branched across multiple different decision criteria in parallel, significantly increasing throughput.

Who it’s for

Developers building AI agents that require high-speed, structured decisions based on unstructured state, and those looking to implement semantic decision-making patterns using open-source models (like Qwen or MiniCPM) on consumer-grade hardware (e.g., an RTX 3090).

Highlights

• Direct Logit Readout: Bypasses token generation entirely, resulting in significantly faster decision times compared to autoregressive JSON generation.
• Shared State Reuse: Enables prefetching a state once to evaluate multiple criteria in parallel, drastically increasing decisions per second.
• Browser-based Execution: Includes a WebGPU demo allowing users to run models directly in the browser.
• Auditable Benchmarks: Provides comprehensive results comparing performance and accuracy against closed services like Jev.