NVIDIA/logits-processor-zoo

What it solves

It addresses the difficulty of getting Large Language Models (LLMs) to strictly follow specific formatting or behavioral instructions during text generation, moving beyond simple grammar enforcement to control the actual content and structure of responses.

How it works

The library provides a collection of "logits processors" that intercept and modify the probability distribution (logits) of the next token before the model samples it. By boosting or diminishing the likelihood of specific tokens, it can force the model to use certain phrases, limit response length, or ensure specific output formats.

Who it’s for

Developers working with LLMs who need precise control over the output format and behavior of their models, particularly those using the transformers, vLLM, or TensorRT-LLM frameworks.

Highlights

• Length Control: Adjusts EOS token likelihood to encourage or discourage shorter answers.
• Prompt-based Citation: Boosts tokens found in the prompt to encourage the model to reference prompt content.
• Phrase Enforcement: Forces the model to include specific phrases (like references or thank-you notes) before finishing.
• Multiple Choice: Constrains the model to answer multiple-choice questions using only the provided options.
• Triggered Phrases: Automatically inserts specific phrases (e.g., Python code blocks) when a specific token is encountered.
• Hallucination Mitigation: Enforces a fallback phrase when the model's token confidence drops below a threshold.
• Time-based Termination: Forces the end of a sentence after a maximum time limit to control generation latency.