Nemotron-pretraining-Indic-v1: NVIDIA and AI4Bharat Release 2.8 Trillion Tokens Across 22 Indian Languages

NVIDIA, IIT Madras, and IIT Madras' AI4Bharat lab announced Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens across 22 Indian languages, releasing soon together with the open, GPU-accelerated data generation pipeline used to build it. The release matters because it targets the central bottleneck in Indic language modeling — insufficient training data — and pairs the corpus with a reusable recipe so teams can generate more data from their own source documents rather than only consuming a fixed dataset.

The 22 languages are those listed in India's Constitution. They are the first languages for 97% of Indians, or nearly 1.2 billion citizens at the 2011 Census.

What exactly was released

The release has two components: the corpus itself and the pipeline that produced it. The pipeline captures the full generation recipe, including prompts, LLM workflows, and quality checks, so developers can adapt it to their own source material.

AI4Bharat, IIT Madras' Indian language research lab, builds open-source datasets and models for Indian languages. Its earlier resources, such as Sangraha, an open pretraining corpus for 22 Indian languages, laid groundwork for open Indic language-model research. The new joint release is described as a substantial expansion of open synthetic Indic training data available to the community.

The corpus brings a broad mix of curated English material — including mathematics, code, and question-answer content — into Indian languages. Every record traces back to a curated source document, and each output is checked against its input, so the synthetic text stays grounded in real information.

Mitesh Khapra, who heads AI4Bharat and was named to the 2025 TIME100 AI list, has named the core problem: Indian language technology trails English because "we do not have enough data for Indian languages." The release answers that with more data plus an open recipe for generating more.

Corpus composition: 2,601.0 billion native-script tokens and 224.6 billion romanized tokens

The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer. Document records include translations and script variants of shared sources; they are not counts of unique English documents.

The corpus spans 34 language-script configurations and splits into two groups:

Native-script and romanized data (12 languages): Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu.
Native-script data only (10 languages): Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi.

In this corpus, native-script describes the writing system: the text is still translated from English. Romanized means Indic-language text written in Latin script, not an English translation. For each of the twelve dual-script languages, romanization covers a subset of the accepted native-script translations.

The distinction matters when constructing a training mixture. Native and romanized records offer different script exposure but can express the same underlying content. The token counts describe the retained corpus, including accepted rewrites, rather than a training run's token budget. Counts also depend on the tokenizer used, so corpus inventory and training-token counts should be read separately.

The twelve-language workflow: translate, assess, repair, romanize

The corpus starts from documents drawn from 15 Nemotron-CC-v2 subsets, covering web text, question-answer material, mathematics, code, and other specialized content. NVIDIA's open-source NeMo Data Designer framework runs translation, quality checks, and targeted repair. Throughout the workflow, each source identifier is retained so developers can trace an output back to its original document.

The twelve-language pipeline has four stages:

Document-level translation. Documents are translated in full so the model can use context across paragraphs, tables, and equations. Gemma-4-31B-it was selected by comparing models on complete documents, checking translation quality and preservation of technical content. Prompts distinguish prose to translate from content to preserve, including code, equations, URLs, and formatting.
Quality assessment. The same model assesses each output against its English source using a rubric inspired by Multidimensional Quality Metrics (MQM), an open framework for classifying translation errors by type and severity. The rubric covers meaning, fluency, terminology, style, and formatting. Only outputs scoring at least 75 out of 100 are retained — a rubric score, not an accuracy percentage.
Targeted repair. For lower-scoring outputs, the assessment feedback guides repairs, which are then reassessed against the same threshold. This process recovered 438.4 billion tokens across native-script and romanized data, already included in the corpus total. Sampled human reviews and tests with deliberately introduced errors helped identify omissions and residual English, informing improvements to prompts and checks.
Romanization. Accepted native-script translations provide the starting point for romanized data. A subset is converted into Latin script and each result is checked against its native-script input. In an initial human comparison, this two-step approach produced fewer hallucinated additions and less residual English than generating romanized text directly from English.

The ten-language workflow: a different translation model and script handling

For the other ten languages, the workflow is adapted to address long-document quality and script-stability issues observed in comparisons. Translation uses Indic Translate, a document-level translation model for English and 22 Indian languages from Bodhan AI, a nonprofit center for AI in education incubated at IIT Madras. Gemma-4-31B-it then assesses each translation against its English source.

Two languages require special handling:

For longer Santali documents, the pipeline translates overlapping windows and reconstructs the document.
For Manipuri, the pipeline translates into Bengali script before converting to Meitei Mayek, its native script.

Eight languages keep the whole-document approach. These workflows retain outputs scoring at least 75 out of 100 and produce native-script data only, without feedback-based rewriting.

What changed after Indic continued pretraining

The Nemotron 3.5 Super VL technical report compares the checkpoint immediately before Indic continued pretraining (CPT) with the checkpoint after that stage. The configured budget was 100 billion training tokens, with 10% (10 billion tokens) allocated to a mixture of synthetic and natural Indic text. The remaining 90% maintained the preceding training mixture's relative proportions.

Scores increased on all five reported Indic knowledge and reasoning benchmarks: MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K. Gains range from 4.98 to 9.82 percentage points. GSM8K excludes Odia because evaluation exceeded the serving context limit.

These are stage-level checkpoints, not a comparison of final model releases, and the comparison reports the change across a training stage that combines synthetic and natural Indic data rather than isolating the synthetic corpus's contribution. Language coverage differs by benchmark. The gains support the combined synthetic-and-natural Indic CPT recipe, while leaving the choice of mixture for other models and languages to further study.

The companion Indic report examines source mixtures, synthetic-data pool size, and script coverage using first-pass accepted data for the twelve-language group. Evaluating the full corpus, including recovered rewrites and the other ten languages, remains a next step for those controlled studies.

Broader Indic capabilities in Nemotron 3.5 Super VL

Beyond the CPT-stage results, the released model was compared with Nemotron 3 Super across Indic task performance and language quality. These results reflect the released model's full training pipeline, including Indic CPT and post-training.

Nemotron 3.5 Super VL scores higher on five of the six task benchmarks shown, spanning knowledge, reasoning, mathematics, code, and translation. Instruction following on IFEval 1.5 dipped 2.04 points. The model also improves across all four IndiVibe dimensions: usefulness, fluency, language and script fidelity, and appropriate verbosity. IndiVibe is Sarvam AI's open benchmark of assistant-style prompts in Indian languages.

The reported IndiVibe figures are preliminary conditional preference scores, not direct head-to-head win rates. Labels show changes in percentage points, except WMT24++, which uses chrF points.

How to load the corpus

Configurations use language, script, and region tags: for example, hi-Deva-IN denotes Hindi in Devanagari, while hi-Latn-IN denotes romanized Hindi. The recommended starting point is to select the languages and scripts a model needs, inspect a sample, and review the dataset card and license before training.

from datasets import get_dataset_split_names, load_dataset

dataset_id = input("Repository ID from the dataset card: ").strip()
config = "hi-Deva-IN"

splits = get_dataset_split_names(dataset_id, config_name=config)
print("Available splits:", splits)

data = load_dataset(
    dataset_id, name=config, split=splits[0], streaming=True
)
print(next(iter(data)))

Hugging Face Datasets supports streaming so readers can inspect records without downloading the entire corpus. Once the release is available, pinning a dataset revision is recommended for reproducible experiments.

For a first training study, the guidance is to compare mixtures at a fixed token budget and record the natural/synthetic and native/romanized shares, then track per-language results alongside English capability retention and language/script quality. When creating held-out splits, records derived from the same source across languages and scripts should be accounted for.

To build a fuller recipe, the corpus can be paired with Nemotron post-training data such as the Chat & Instruction or Post-Training Blends collections. For India-specific synthetic data, NeMo Data Designer can be seeded with Nemotron-Personas-India, synthetic personas in English and Hindi grounded in real-world distributions for India.

How to adapt the generation recipe

The open-source Nemotron Bharat translation plugin for NeMo Data Designer provides translation and evaluation prompts, optional feedback-based rewriting, document windowing, and script-remapping support. Its configuration reference covers choosing models and languages, adapting prompts to a domain, and setting rewrite parameters. An accompanying developer note explains the workflow and engineering choices.

The recommended approach is to start with a small sample of your own source documents, inspect translation quality and preservation of code, mathematics, and formatting, then refine prompts and checks with native-language reviewers before scaling up. Generation and corpus acceptance are treated as separate steps: the plugin returns candidates and scores, while the filtering policy determines which records enter the training corpus.

Scope and limitations

Translation broadens access to source material but does not replace text originally written in Indian languages or guarantee coverage of local knowledge and culture. NVIDIA recommends combining this corpus with appropriately sourced natural Indic text and evaluating the resulting mixture for the intended use case.

Quality assessment is primarily automated, complemented by sampled human review whose coverage varies across languages. In the twelve-language workflow, the translator also serves as the judge. Retained text can still contain errors, and the ten-language workflow needs broader native-speaker validation. Source-ID checks remove processing duplicates; they are not semantic deduplication of similar content.

Alongside the corpus, AI4Bharat at IIT Madras, with support from NVIDIA, is building Indic-LLM-Eval-Suite, an open and evolving collection of benchmarks for Indic model capabilities that helps model builders assess both task performance and language quality.