NeoMME: Efficient Multimodal-native and Multilingual Encoder
Hugging Face has introduced NeoMME, a family of multilingual multimodal encoders available in 260M and 800M parameter versions. Unlike traditional generative visual language models (VLMs) that rely on separate pretrained vision towers and causal language models, NeoMME uses a single bidirectional Transformer to process both text tokens and raw image patches, trained from scratch using a masked discrete-diffusion objective.
Multimodal-Native Architecture
NeoMME eliminates the parameter and compute overhead associated with separate vision encoders and projectors by processing images and text through the same computational path. This design simplifies pretraining, fine-tuning, and serving across modalities.
Core Technical Specifications
- Unified Processing: Text uses factorized token embeddings, while images are divided into 32×32 non-overlapping patches and projected via a small MLP.
- Dynamic Resolution: The model preserves the aspect ratio and size of images, allowing it to allocate more tokens to high-resolution, information-dense document pages.
- Long Context Window: A context length of 16,384 tokens supports up to two 4K UHD images. The architecture utilizes symmetric sliding-window attention for most layers, with global attention every sixth layer and in the final layer.
- Modern Encoder Stack: The model incorporates grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs.
- Multilingual Support: A custom BPE tokenizer with a 131k-token vocabulary was trained from scratch on multilingual text, code, mathematics, and image transcripts.
Pretraining Objective
NeoMME is pretrained as a discrete masked-diffusion text denoiser. For multimodal examples, image patches remain visible while the model reconstructs masked text. By applying corruption rates between 0.3 and 1, the model is forced to rely on visible image evidence rather than language-only shortcuts, grounding the text in the visual data.
NeoMME-Retriever for Visual Document Retrieval
To evaluate the backbone, Hugging Face developed NeoMME-Retriever, which fine-tunes the model for visual document retrieval using the ColPali page-image methodology. This approach ranks document page screenshots directly, bypassing OCR and preserving visual cues like layout, charts, and font styles.
Dual-Head Design
NeoMME-Retriever employs two jointly trained heads to provide flexibility in retrieval infrastructure:
- Dense Head: Uses mean pooling to create a single normalized vector, compatible with approximate nearest-neighbor (ANN) techniques for fast, compact retrieval.
- Late-Interaction Head: Projects each token or patch into a 128-dimensional normalized vector, preserving local matches between query tokens and image regions for higher precision.
Performance and Benchmarks
On the ViDoRe v3 benchmark, NeoMME-Retriever-260M achieved an nDCG@10 of 0.523, the highest score for any model under 800M parameters. It performs within 0.002 nDCG@10 of the much larger ColQwen2.5 (3.75B parameters) while using approximately 14× fewer parameters.
| Model | Params | ViDoRe v3 (nDCG@10) | ViDoRe v2 (nDCG@5) | ViDoRe v1 (nDCG@5) |
|---|---|---|---|---|
| ColModernVBERT | 250M | 0.261 | 0.407 | 0.806 |
| NeoMME-260M | 260M | 0.523 | 0.522 | 0.860 |
| ColSmol-500M | 500M | 0.340 | 0.455 | 0.825 |
| NeoMME-800M | 800M | 0.556 | 0.559 | 0.874 |
| Vultron Flash | 850M | 0.565 | 0.604 | 0.882 |
| ColQwen2.5-v0.2 | 3.75B | 0.524 | 0.601 | 0.895 |
Efficiency and Optimization
Index Storage Compression
Late-interaction embeddings for high-resolution images can be storage-intensive (averaging 1.5 MB per document). NeoMME employs two methods to reduce this footprint:
- Hierarchical Token Pooling: Clusters similar document vectors and replaces them with their mean.
- Asymmetric Quantization: Quantizes document embeddings to int8 or binary while keeping query embeddings at higher precision.
Using a pooling factor of 8 and binary documents, storage is reduced from 1.5 MB to 6 kB per page (a 255× reduction) while retaining over 95% of the baseline nDCG@10.
Inference Throughput
NeoMME-Retriever-260M demonstrates significant speed advantages during corpus indexing. At a 2048×2048 input size on an NVIDIA L40S GPU, it encodes approximately 51 pages per second, nearly double the throughput of ColModernVBERT (26 pages per second).
Visual RAG Integration
NeoMME-Retriever serves as the first stage of a visual Retrieval-Augmented Generation (RAG) pipeline. Instead of retrieving text chunks, the system retrieves original page images and passes them to a visual language model (VLM). This allows the VLM to utilize the original layout, diagrams, and tables that are often lost during text extraction.
All NeoMME checkpoints are released under the Apache 2.0 license and are integrated into the Hugging Face Transformers library.