Ollama Multimodal Engine Release

Ollama has launched a new multimodal engine designed to make vision models first-class citizens within its ecosystem. This update improves the reliability and accuracy of local inference and establishes a foundation for future support of modalities such as speech, image generation, video generation, and expanded tool support.

Supported Multimodal Models

Ollama's new engine enables the local execution of several high-performance vision models, including:

Meta Llama 4 (including the 109B parameter Mixture-of-Experts model, Llama 4 Scout)
Google Gemma 3
Qwen 2.5 VL
Mistral Small 3.1

Key Technical Capabilities

General Understanding and Reasoning

The new engine supports complex visual reasoning and multi-image analysis. For example, Llama 4 Scout can analyze video frames to identify locations (such as the Ferry Building in San Francisco) and provide location-based reasoning, such as calculating distances to other landmarks.

Gemma 3 supports multi-image inputs, allowing users to input several images simultaneously or via follow-up prompts to identify common elements across images or reason about the relationship between different visual subjects.

Document Scanning and OCR

Qwen 2.5 VL is utilized for high-accuracy character recognition and document scanning. This includes the ability to understand and translate vertical Chinese spring couplets into English.

Architectural Improvements

Model Modularity

To improve reliability and simplify integration, Ollama has moved toward a self-contained model architecture. Previously, relying on ggml/llama.cpp meant that text decoders and vision encoders were split into separate models executed independently, requiring model-specific logic in the orchestration layer.

In the new engine, each model is fully self-contained and exposes its own projection layer aligned with its training. This isolation prevents the need for patching multiple files or using cascading if statements, allowing model creators to implement their code without risking regressions in other models.

Inference Accuracy

Ollama has implemented metadata processing to handle large images that produce a high volume of tokens. By managing causal attention and determining the optimal boundaries for splitting image embeddings into batches, Ollama ensures that image processing follows the model's original design and training, preventing the degradation of output quality that occurs when images are split incorrectly.

Memory Management and Optimization

Ollama has introduced several memory-specific optimizations to increase performance and reduce resource consumption:

Image Caching: Processed images are cached to accelerate subsequent prompts and are maintained in the cache as long as they are in use.
Hardware Integration: Ollama collaborates with hardware manufacturers (NVIDIA, AMD, Qualcomm, Intel) and Microsoft to detect hardware metadata for better memory estimation.
KV Cache Optimizations: Causal attention is configured at the individual model level. For instance, Gemma 3 utilizes sliding window attention to optimize context length allocation and increase concurrency.
Llama 4 Support: For Llama 4 Scout and Maverick, Ollama implemented chunked attention, attention tuning for longer context, specific 2D rotary embedding, and Mixture-of-Experts (MoE) support.

Future Roadmap

Ollama is developing further enhancements to the platform, including:

Support for longer context sizes
Integration of thinking and reasoning capabilities
Tool calling with streaming responses
Enabling "computer use" capabilities