NVIDIA/recsys-examples

Examples for Recommenders - easy to train and deploy on accelerated infrastructure.

What it solves

This project provides optimized implementations and GPU-accelerated components for large-scale recommender systems, focusing on reducing latency and increasing throughput for both training and inference of ranking and retrieval models.

How it works

The repository is split into high-level examples and low-level GPU libraries:

  • Examples: Implements advanced recommender architectures like HSTU (Hierarchical Sequential Transformer Unit) for ranking and retrieval, and SID-GR (Semantic ID Generative Recommender) for generative retrieval using hierarchical semantic-ID prediction.
  • GPU Libraries: Provides specialized infrastructure such as DynamicEmb for model-parallel dynamic embedding tables with flexible eviction policies (LRU/LFU), a RecSys KVCache Manager for user-ID-based KV-cache reuse, and optimized beam-search decode attention kernels.
  • Optimizations: Leverages NVIDIA-specific hardware acceleration (Blackwell, SM8x, SM90), Triton Inference Server, CUDA graphs, and AOTInductor for C++ deployment.

Who it’s for

ML engineers and researchers building industrial-scale recommender systems who need to optimize GPU utilization, manage massive embedding tables, and deploy generative retrieval models at scale.

Highlights

  • Generative Recommendation: Support for SID-GR models with optimized beam search and KV-cache management.
  • C++ Deployment: End-to-end HSTU inference workflow using AOTInductor for native C++ replay.
  • Dynamic Embedding Management: Advanced embedding tables with GPU/host hash-table storage and compound eviction policies.
  • High-Performance Inference: Integration with Triton Inference Server and paged GPU KV caches to hide onload/offload latency.

Related

  • Project
    NVIDIA-Merlin/HugeCTR

    A GPU-accelerated recommender framework designed for the efficient training and inference of large deep learning models with massive embedding tables.

  • Project
    NVIDIA-Merlin/Merlin

    NVIDIA Merlin is an open-source library suite that accelerates the end-to-end creation of large-scale recommender systems using GPU acceleration for feature engineering, training, and inference.

  • Project
    cfregly/ai-performance-engineering

    A comprehensive collection of code and resources for optimizing AI systems, focusing on GPU tuning, distributed training, and high-throughput inference scaling.

  • Project
    NVIDIA/cuvs

    A GPU-accelerated library for vector search and clustering, providing high-performance approximate nearest neighbors implementations for semantic search and data mining.

  • Project
    meta-recsys/generative-recommenders

    A framework for generative recommendation systems that reformulates recommendation as a generative modeling problem to overcome compute bottlenecks and scale to trillion-parameter models.