Ollama 0.31: Faster Gemma 4 Performance via Multi-Token Prediction

Ollama 0.31 implements multi-token prediction (MTP) for Gemma 4 on Apple Silicon, resulting in token generation speeds that are nearly 90% faster on average across coding-agent benchmarks. This performance increase is enabled by default and does not change the model's output.

Multi-Token Prediction (MTP) Mechanism

Gemma 4 utilizes a small, fast draft model that runs alongside the main model to propose the next several tokens. The main model then verifies these proposals in a single pass, committing multiple tokens for the cost of one whenever the draft model's predictions are correct.

This approach is particularly effective for code generation, which often contains predictable patterns such as closing brackets, repeated identifiers, and boilerplate. Consequently, coding agents that continuously call the model to read files and execute tools become significantly more responsive.

Technical Implementation

The speedup in Ollama 0.31 is achieved through three primary technical optimizations: auto-tuning draft length, engine-level speculative decoding, and GPU kernel optimizations.

Auto-Tuning Draft Length

Because the ideal number of tokens to draft varies based on the model, quantization, hardware, and the predictability of the text, Ollama determines the draft length at runtime. The system tracks proposal acceptance rates and verification pass durations to select the length that maximizes tokens per second. If proposals are consistently rejected, the system automatically reverts to standard one-at-a-time decoding to ensure that speculation does not degrade performance.

Speculative Decoding Engine

The generation process follows a specific sequence executed entirely on the GPU to avoid CPU round-trips:

Drafting: The draft model predicts a sequence of tokens.
Sampling and Verification: The main model verifies the entire run of proposals in a single pass.
Commitment: Accepted tokens are kept. For rejected tokens, the engine uses a rollback point recorded before each proposal to rewind to the last accepted token without recomputing earlier state.

MLX Kernel Optimizations

Verification is the most computationally expensive part of the process. Because draft batches are typically small (2 to 8 tokens), they fall between the size of a single-token decode and a large-batch prefill, making standard matrix multiplication kernels inefficient.

Ollama contributed a specialized kernel to MLX that reads and unpacks each block of weights once and reuses them across the entire batch. On an M5 Max using nvfp4, this optimization makes Gemma 4's largest matrix multiplications 2× to 2.5× faster by removing redundant work while maintaining identical computation.

Benchmarks and Availability

Performance was measured using the Aider polyglot benchmark, which simulates real programming tasks with a coding agent. The results showed a nearly 90% increase in generation speed for Gemma 4 12B (nvfp4) on an M5 Max.

To use these improvements, users must download Ollama 0.31 or later for macOS. Users who previously downloaded Gemma 4 should re-pull the model using the following command:

ollama pull gemma4:12b-mlx