Domain-Specific Embedding Fine-Tuning with NVIDIA Nemotron – Under a Day
TL;DR: NVIDIA and Hugging Face released a six‑step, single‑GPU pipeline that fine‑tunes the 1‑billion‑parameter Llama‑Nemotron‑Embed‑1B‑v2 model on synthetic domain data in under a day, delivering >10 % gains in Recall@10/NDCG@10 and up to 26 % improvement on real‑world enterprise datasets.
Overview – Why Domain‑Specific Embedding Fine‑Tuning Matters
General‑purpose embedding models excel at internet‑scale semantics but miss the fine‑grained distinctions required for contracts, manufacturing logs, or internal taxonomies. Fine‑tuning with domain‑specific data bridges this gap, improving retrieval quality in Retrieval‑Augmented Generation (RAG) pipelines. The new recipe automates data creation, hard‑negative mining, multi‑hop handling, training, evaluation, and deployment, all with a single NVIDIA Ampere‑class GPU.
Quick Access Links
- Embedding model:
nvidia/llama-nemotron-embed-1b-v2 - Code repository: https://github.com/NVIDIA-NeMo/Nemotron/tree/main/src/nemotron/recipes/embed
- Synthetic dataset (NVDocs): https://huggingface.co/datasets/nvidia/Retrieval-Synthetic-NVDocs-v1
Integrated Open‑Source Components
| Component | Role |
|---|---|
| NeMo Data Designer | Generates synthetic question‑answer pairs from raw documents |
| NeMo Automodel | Trains the bi‑encoder embedding model |
| BEIR | Provides a standardized evaluation benchmark |
| NeMo Export‑Deploy | Converts checkpoints to ONNX/TensorRT |
| NVIDIA NIM | Serves the model via an OpenAI‑compatible embeddings endpoint |
Prerequisites
- A directory of domain documents (
.txt,.md, etc.) - A free NVIDIA API key from https://build.nvidia.com/
- An NVIDIA Ampere GPU or newer with ≥ 80 GB VRAM (tested on A100‑80GB and H100‑80GB)
Step‑by‑Step Pipeline
1️⃣ Generate Synthetic Training Data
The pipeline uses the LLM nvidia/nemotron-3-nano-30b-a3b to read each document and produce high‑quality QA pairs without any manual labeling.
nemotron embed sdg -c default corpus_dir=./data/my_domain_docs
The four‑stage SDG pipeline (implemented in NeMo Data Designer) creates questions of varying complexity (1‑3 hops) and assigns a quality score. Only pairs above a configurable threshold are kept for training.
2️⃣ Mine Hard Negatives
Hard negatives are non‑relevant passages that the base model ranks close to the positive answer. Mining proceeds as follows:
- Embed all queries and corpus passages with the base model.
- Compute similarity scores.
- Mask out the true positives.
- Apply a 95 % margin filter to avoid false negatives.
- Select the top‑k (default 5) highest‑scoring non‑positives as hard negatives.
nemotron embed prep -c default
The command also splits the data (80 % train, 20 % test) and unrolls multi‑hop questions into independent (query, positive) examples, preserving the same hard negatives for each.
3️⃣ Understand Multi‑Hop Questions
Multi‑hop queries (2‑3 hops) require the model to retrieve several related passages. By unrolling each hop into separate training instances, the model learns to treat all relevant documents as positives, improving its ability to surface complete answer sets for complex user queries.
4️⃣ Fine‑Tune the Bi‑Encoder
Training uses contrastive loss with a temperature of 0.02, forcing the model to sharply separate positives from hard negatives.
nemotron embed finetune -c default
Key hyperparameters (default values are tuned for the example dataset):
| Parameter | Default |
|---|---|
| Epochs | 3 (1–2 epochs recommended for larger corpora) |
| Learning rate | 1e‑5 |
| Warmup steps | 5 (≈ 5‑10 % of total steps) |
| Global batch size | 128 |
| Passages per query | 5 (1 positive + 4 hard negatives) |
5️⃣ Evaluate Retrieval Gains
Evaluation runs the BEIR framework on the held‑out test set and reports nDCG, Recall, Precision, and MAP at k = 1, 5, 10, 100.
nemotron embed eval -c default
Synthetic NVDocs results (base → fine‑tuned):
- NDCG@10: 0.555 → 0.616 (+10.9 %)
- Recall@10: 0.630 → 0.693 (+10.0 %)
- Similar gains observed across all k values.
Real‑world Atlassian case: Fine‑tuning on a public JIRA dataset raised Recall@60 from 0.751 to 0.951, a 26.7 % improvement, using a single A100‑80GB GPU.
Troubleshooting
- Low synthetic quality → improve document formatting or use a stronger LLM.
- Insufficient data → add more source documents and re‑run SDG.
- Over‑fitting → reduce epochs to 1‑2 or raise the quality threshold.
- Sub‑optimal learning rate → try 0.5× or 2× the default.
6️⃣ Export and Deploy
Convert the fine‑tuned checkpoint to ONNX (opset 17) and optionally compile a TensorRT engine for maximum throughput.
nemotron embed export -c default # ONNX only
nemotron embed export -c default export_to_trt=false
nemotron embed export -c default quant_cfg=fp8 # FP8 quantization
Deploy the model with NVIDIA NIM, which exposes an OpenAI‑compatible /v1/embeddings endpoint.
nemotron embed deploy -c default
Example request:
curl -X POST http://localhost:8000/v1/embeddings \
-H "Content-Type: application/json" \
-d '{"input": ["What cooling is needed for 8 H100 GPUs in a 2U chassis?"], "model": "custom", "input_type": "query"}'
A built‑in NIM accuracy check re‑evaluates the deployed service against BEIR to ensure conversion did not degrade performance.
End‑to‑End Command Summary
# 1. Synthetic data generation
nemotron embed sdg -c default corpus_dir=./data/my_docs
# 2. Data preparation (split, hard‑negative mining, unroll)
nemotron embed prep -c default
# 3. Fine‑tune the embedding model
nemotron embed finetune -c default
# 4. Evaluate base vs. fine‑tuned checkpoint
nemotron embed eval -c default
# 5. Export to ONNX/TensorRT
nemotron embed export -c default
# 6. Deploy with NVIDIA NIM
nemotron embed deploy -c default
Resource and Time Estimates
| Stage | GPU Needed? | Approx. Time (single A100‑80GB) |
|---|---|---|
| Synthetic Data Generation | No (API) | ~1 hour (depends on corpus size) |
| Data Preparation (hard‑negative mining) | Yes (≈ 40 GB VRAM) | ~5 min |
| Fine‑Tuning | Yes (80 GB VRAM) | ~1 hour |
| Evaluation | Yes (≈ 40 GB VRAM) | ~5 min |
| Export | Yes (≈ 40 GB VRAM) | ~5 min |
| Deployment | Yes (≈ 40 GB VRAM) | ~5 min |
| Total: < 24 hours; for a modest corpus (~500 documents) the whole flow completes in 2–3 hours. |
Practical Takeaway
The recipe demonstrates that domain‑adapted embeddings are no longer a multi‑week, multi‑GPU effort. By leveraging synthetic data generation, hard‑negative mining, and NVIDIA’s optimized tooling, practitioners can achieve substantial retrieval improvements—often >10 % on standard metrics and >25 % on enterprise workloads—using a single GPU and less than a day of compute.
Try It Yourself
Clone the repositories, obtain an NVIDIA API key, and point the pipeline at your own document collection. The ready‑made nvidia/Retrieval-Synthetic-NVDocs-v1 dataset lets you experiment immediately. Contributions and stars on the Nemotron, NeMo Data Designer, and NeMo Automodel repos are welcomed.