NVIDIA LogitsProcessorZoo: Controlling Language Model Generation with Modular Logits Processors

What Are Logits in Language Models?

Logits are the raw, unnormalized scores generated by language models for each token in their vocabulary, which are turned into probabilities via the softmax function to guide next-token selection.

The generation process works as follows: the model outputs logits for each token; applying softmax converts them into a probability distribution; sampling or argmax selects the next token. The 🤗 Transformers library’s generate() method handles these steps automatically.

Why Process Logits?

Processing logits lets developers modify the probability distribution before token selection to enforce constraints, avoid overgeneralization, and align model outputs with specific task requirements.

Raw logits may lack needed constraints, produce generic responses, or misalign with tasks such as length limits, phrase inclusion, or multiple-choice formatting.

NVIDIA's LogitsProcessorZoo

NVIDIA's LogitsProcessorZoo is a collection of modular logits processors compatible with Hugging Face Transformers that enable control over sequence length, prompt citation, forced phrase inclusion, and multiple-choice answer selection.

The library is installable via pip install logits-processor-zoo and works with the generate() method through a LogitsProcessorList.

GenLengthLogitsProcessor

The GenLengthLogitsProcessor controls the length of generated sequences by adjusting the likelihood of the end-of-sequence (EOS) token.

Increasing the boost factor makes the model more likely to emit EOS sooner, yielding shorter outputs; decreasing it (or using a negative value) discourages EOS, yielding longer outputs. The processor can also be configured to complete sentences before stopping.

Example usage from the post:

[GenLengthLogitsProcessor(runner.tokenizer, boost_factor=0.1, p=2, complete_sentences=True)]

produces a short story, while

[GenLengthLogitsProcessor(runner.tokenizer, boost_factor=-10.0, p=0, complete_sentences=False)]

produces a longer continuation.

CiteFromPromptLogitsProcessor

The CiteFromPromptLogitsProcessor boosts or diminishes tokens that appear in the prompt to encourage the model to generate content that closely reflects the input.

This is useful for tasks such as question answering over a passage or summarizing with specific details.

Example usage:

[CiteFromPromptLogitsProcessor(runner.tokenizer, example_prompts, boost_factor=5.0)]

with a user review prompt yields a response that repeats the review’s opinion about price.

ForceLastPhraseLogitsProcessor

The ForceLastPhraseLogitsProcessor forces the model to include a user‑specified phrase before ending its output.

This helps maintain consistent formatting for citations, reports, or any structured output that must end with a particular string.

Example usage:

[ForceLastPhraseLogitsProcessor(phrase, runner.tokenizer, batch_size)]

where phrase is set to "\n\nReferences:" adds a reference block, or to "\n\nThanks for trying our RAG application!" adds a closing note.

MultipleChoiceLogitsProcessor

The MultipleChoiceLogitsProcessor guides the model to select exactly one of the provided choices for a multiple‑choice question, suppressing all other tokens.

This ensures the output conforms to a strict answer format, which is valuable for quizzes, surveys, or decision‑making systems.

Example usage:

MultipleChoiceLogitsProcessor(
    runner.tokenizer,
    choices=["0", "1", "2", "3"],
    delimiter=\n\n" .
\)

with a prompt asking about phone features returns only the selected option, e.g., "1).

Wrapping Up

Logit processors provide the flexibility to control language model outputs effectively, making them invaluable for scenarios where precision, adherence to constraints, or task‑specific behavior is critical.

Resources for further exploration include the Transformers generation guide, generation strategies documentation, the LogitsProcessor API reference, and the NVIDIA LogitsProcessorZoo repository with examples and use cases.

With these tools, developers can refine AI workflows to meet precise generation requirements.

Sources