Evaluation: Retrieval Quality, Agentic Efficiency, and Deployment Tradeoffs ：#evaluation-retrieval-quality-agentic-efficiency-and-deployment-tradeoffs RTEB Leadership and Strong Gains Across Retrieval Benchmarks ：#rteb-leadership-and-strong-gains-across-retrieval-benchmarks Why Better Retrieval Matters for Agents ：#why-better-retrieval-matters-for-agents Scaling Retrieval with NVFP4 on Blackwell ：#scaling-retrieval-with-nvfp4-on-blackwell Day 0 Performant NIM ：#day-0-performant-nim How We Built the Nemotron 3 Embed Models ：#how-we-built-the-nemotron-3-embed-models Scaling Down to 1B ：#scaling-down-to-1b Enterprise Partner Evaluations ：#enterprise-partner-evaluations Getting Started ：#getting-started Retrieval is critical in multi-step agentic workflows where poor retrieval can cause agents to fetch irrelevant context, re-query, waste token budget, and carry noise into later reasoning steps.
Today, we are releasing NVIDIA Nemotron 3 Embed ：https://huggingface.co/collections/nvidia/nemotron-3-embed, a collection of open and commercially available embedding models designed to improve retrieval quality while giving developers practical deployment options for production-scale RAG, agentic retrieval, code retrieval, and agent memory.
The collection includes three open models that achieve state-of-the-art retrieval across the accuracy-efficiency curve, led by an 8B model that tops the RTEB leaderboard and efficient 1B variants built for production-scale deployment:
Table 1. Nemotron 3 Embed Model Usability and Deployment Matrix.
Beyond the RTEB result, Nemotron 3 Embed introduces a production-ready feature set for enterprise retrieval deployments:
We evaluate Nemotron 3 Embed across three dimensions: retrieval quality, downstream agentic efficiency, and deployment tradeoffs. The 8B model establishes the model collection’s quality ceiling, while the 1B BF16 and NVFP4 variants bring the same retrieval-focused design to lower-cost and higher-throughput deployment settings.
We first evaluated the models on RTEB：https://huggingface.co/blog/rteb, where Nemotron-3-Embed-8B-BF16 ranks #1. We also tested these models across ViDoRe V3：https://huggingface.co/blog/QuentinJG/introducing-vidore-v3 Text, and MMTEB：https://arxiv.org/abs/2502.13595 Retrieval and LongEmbed：https://github.com/dwzhu-pku/longembed using average NDCG@10.
Figure 2. Retrieval accuracy using average NDCG@10 across RTEB, ViDoRe V3 Text, MMTEB Retrieval and LongEmbed, comparing the Nemotron 3 Embed models with prior-generation Nemotron baselines.
To evaluate retrieval in an agentic setting, we use a search agent：https://huggingface.co/blog/nvidia/nemo-retriever-agentic-retrieval powered by Nemotron 3 Ultra：https://github.com/NVIDIA-NeMo/Nemotron/tree/main/usage-cookbook/Nemotron-3-Ultra and vary the embedding model used by the retrieval system. Better retrieval can return relevant evidence earlier, helping the agent avoid repeated searches, unnecessary reasoning turns, and extra context inspection. We compare average retrieval accuracy with estimated downstream agentic token cost per query across ViDoRe V3, BRIGHT：https://brightbenchmark.github.io/, and BrowseComp-Plus：https://huggingface.co/spaces/Tevatron/BrowseComp-Plus.
Figure 3. Average retrieval accuracy versus downstream agentic token cost per query across ViDoRe V3, BRIGHT, and BrowseComp-Plus.
Evaluation note: The search agent uses Nemotron 3 Ultra. Downstream token cost is estimated from Nemotron 3 Ultra input/output token counts using the GPT-5.5 pricing formula.
Figure 3 shows that stronger retrieval reduces downstream agentic token cost. More accurate retrievers return relevant evidence earlier, which helps agents complete tasks with fewer repeated searches and fewer reasoning turns. In these evaluations, the Nemotron 3 Embed models improve the agentic retrieval frontier, with the 8B model delivering both the highest average retrieval accuracy and the lowest estimated downstream token cost across ViDoRe V3, BRIGHT, and BrowseComp-Plus.
For high-throughput deployments, teams often choose smaller embedding models to meet latency and cost targets. Nemotron-3-Embed-1B-NVFP4 is designed to narrow the gap between serving efficiency and retrieval quality by using native NVFP4 acceleration on NVIDIA Blackwell architectures：https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture/. The model quantizes the weights and activations of linear layers to NVFP4 for efficient inference, and uses Quantization-Aware Distillation (QAD) to help recover accuracy for long input sequences.
Figure 4. ViDoRe V3 retrieval accuracy versus serving efficiency, comparing Nemotron-3-Embed-1B-NVFP4 with selected smaller open embedding baselines, including Qwen3-Embedding-0.6B and EmbeddingGemma-300M.
For production-scale retrieval systems, the serving stack also needs to preserve that efficiency under real request loads, across different input sequence lengths and hardware targets. To make Nemotron 3 Embed performant at enterprise scale today, we are also releasing an optimized NVIDIA NIM microservice for the 1B model. As shown in Figure 5, the Rust-based Nemotron 3 Embed NIM matches or outperforms the vLLM checkpoint on NVIDIA GB200 and RTX PRO 6000 GPUs across ISLs of 256 & 1024.
Figure 5. Nemotron 3 Embed NIM serving performance compared with the vLLM checkpoint on NVIDIA GB200 and RTX PRO 6000 GPUs.
Nemotron-3-Embed-8B-BF16 adapts the Ministral-3-8B-Instruct-2512：https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512 backbone by converting its causal decoder into a bidirectional encoder for full-sequence retrieval. The model is trained with contrastive pre-training on a blend of web-sourced and synthetic text pairs, then fine-tuned on curated multilingual retrieval datasets across domains such as legal, finance, medical, business, and education. This 8B model serves as the flagship embedding model, while earlier 8B teacher checkpoints from the same development line were used to distill the efficient 1B variants.
The 1B model is not a small retriever trained from scratch. We first applied the bidirectional adaptation recipe to the Ministral-3-3B-Instruct-2512：https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512 backbone to establish a 3B retriever base, then compressed it through two-rounds of structured pruning and distillation.
First, the 3B parent model was compressed to a 2B intermediate footprint using NVIDIA ModelOpt：https://github.com/NVIDIA/Model-Optimizer’s mcore_minitron Neural Architecture Search engine. The NAS pipeline searched across hidden width, FFN size, attention heads, and depth under a strict parameter budget to identify an efficient architecture for retrieval workloads.
The resulting 2B intermediate model was then distilled from an 8B teacher checkpoint to recover ranking accuracy. We used a combined cosine distance loss and mean squared error loss on a multilingual, in-domain retrieval data blend to align the student’s embeddings with the teacher.
Figure 6. The pruning and distillation pipeline compresses the retriever from a 3B base to the final 1B production model.
This same sequence, ModelOpt structured pruning followed by 8B teacher distillation, was repeated a second time to compress the 2B intermediate model down to the final 1.14B embedding model. Final training used a progressive two-stage context-scaling schedule:
The following table summarizes the core technical specifications and deployment targets for the Nemotron 3 Embed models:
Table 2. Architectural specifications and core inference configurations for the Nemotron 3 Embed models.
Enterprise ISVs, AI-native companies, data platform companies, and memory providers are already evaluating Nemotron 3 Embed across agentic retrieval, agent memory, code retrieval, and production inference workflows.
NVIDIA Nemotron 3 Embed is released with open weights and open-source training recipes, giving organizations full control over how retrieval models are customized and deployed for production AI applications.
Developers can get started using the deployment option that best fits their workflow:
For workloads requiring domain adaptation or footprint reduction, we have also open-sourced NVIDIA NeMo AutoModel：https://github.com/nvidia-nemo/automodel training recipes:
For example, on the NV Docs：https://huggingface.co/datasets/nvidia/Retrieval-Synthetic-NVDocs-v1 evaluation, fine-tuning Nemotron-3-Embed-1B-BF16 improved NDCG@10 from 56.7% to 63.3% (+11.6%) and Recall@5 from 56.1% to 62.8% (+11.9%).
Whether you are building enterprise search, production RAG, agent memory, code retrieval, or agentic AI systems, NVIDIA Nemotron 3 Embed provides flexible deployment options—from open-source models on Hugging Face to fully managed AI Cloud platforms and NVIDIA NIM microservices.
Explore the models, deploy them on your preferred platform, and let us know what you are building.
Fair and Disentangled Evaluation of Deep-Research Agents
Congratulations on the LMEB SOTA! 🎉
The results merged in https://github.com/embeddings-benchmark/results/pull/617：https://github.com/embeddings-benchmark/results/pull/617 show outstanding LMEB performance:
Nemotron-3-Embed-8B-BF16: 64.4, a new overall SOTA. Nemotron-3-Embed-1B-BF16: 61.5, a new SOTA at the ~1B scale.
The LMEB leaderboard：https://mteb-leaderboard.hf.space/benchmark/LMEB appears to be awaiting synchronization.
It's live now! Thank you for evaluating our models there!
扫码直达文章详情