NVIDIA Llama Nemotron Nano VL Release

NVIDIA has announced the release of Llama Nemotron Nano VL, an 8B Vision Language Model (VLM) designed for intelligent document processing (IDP) and optical character recognition (OCR). The model is optimized for extracting and understanding information from complex documents, including invoices, receipts, contracts, and financial statements, and is now available on the Hugging Face Hub.

Technical Architecture and Innovations

Llama Nemotron Nano VL is built upon the Llama-3.1-8B-Instruct language model and the C-RADIOv2-VLM-H Vision Transformer (ViT). This combination allows the model to handle a wide variety of visual elements, including charts, graphs, and complex layouts.

Vision Foundation

The C-RADIOv2-VLM-H ViT serves as the vision backbone, enabling high-resolution processing of documents. Key innovations include:

  • High-Resolution Tiling: A design that dynamically aggregates encoded patch features to support high-resolution input without sacrificing spatial continuity or computational efficiency.
  • Multi-Resolution Training: C-RADIO was trained on multi-resolution data using multiple distillation techniques and multiplicative noise to improve generalization.
  • Aspect Ratio Flexibility: The model can process documents with arbitrary aspect ratios while preserving local detail and global context, which is critical for analyzing small fonts and multi-column layouts.

Training Methodology

The model underwent a two-stage training regimen:

  1. Pre-training: Focused on cross-modal alignment between language and vision domains using a Multi-Layer Perceptron (MLP) connector. This stage utilized a dataset of approximately 1.5 million samples, comprising public, synthetic, and internally curated data.
  2. Supervised Fine-Tuning (SFT): The model was trained end-to-end on a blend of OCR-focused data, including tasks such as predicting reading order, reconstructing markdown formatting, and parsing mathematical formulas in LaTeX. To ensure robustness, NVIDIA applied affine and photometric augmentations to document images.

Capabilities and Performance

Llama Nemotron Nano VL excels in visual reasoning and text recognition, demonstrating industry-leading performance on the OCRBench v2 benchmark. It also achieves strong results on ChartQA and AI2D benchmarks.

Key Functional Capabilities

  • Text and Table Extraction: High accuracy in spotting and extracting text and tabular data from documents.
  • Element Parsing: Accurate identification of critical elements such as tables, charts, and images.
  • Grounding: Support for bounding box coordinates in normalized space (0-1000) for both queries and outputs, enabling text-referring and interpretability.
  • Multimodal Output Formats: Ability to extract tables in LaTeX, HTML, or markdown formats based on the prompt.

Enterprise Use Cases

Due to its layout-aware reasoning and efficiency on a single GPU, the model is targeted at large-scale enterprise automation in several domains:

  • Finance and Accounting: Automating line item and total extraction from invoices and receipts for ERP integration.
  • Legal and Compliance: Reviewing contracts for key clauses and analyzing identity documents (passports, IDs) for KYC processes.
  • Healthcare: Extracting patient data and claim information from medical records and insurance forms.

Deployment and Customization

Llama Nemotron Nano VL is available via the NVIDIA NIM API and for download on Hugging Face. Developers can further post-train or fine-tune the model using NVIDIA NeMo to adapt it to specific proprietary datasets.

Sources