huggingface/nanotron

Minimalistic large language model 3D-parallelism training

What it solves

Nanotron simplifies the complex process of pretraining transformer models, providing a scalable and high-performance framework that allows users to train models on custom datasets without needing to build the underlying distributed training infrastructure from scratch.

How it works

It provides a flexible API and a set of optimized training scripts that leverage advanced distributed training techniques. The library implements 3D parallelism (combining Data, Tensor, and Pipeline parallelism) and expert parallelism for Mixture-of-Experts (MoE) models to distribute the workload across multiple GPUs and nodes efficiently.

Who it’s for

It is designed for researchers and engineers who need to pretrain large-scale language models (LLMs) or other transformer-based architectures and require a tool that is both easy to use and capable of scaling to massive compute clusters.

Highlights

  • 3D Parallelism: Supports Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP).
  • MoE Support: Includes expert parallelism for Mixture-of-Experts models.
  • Optimization Techniques: Features ZeRO-1 optimizer, FP32 gradient accumulation, and custom module checkpointing for memory efficiency.
  • Scaling Tools: Implements spectral µTransfer parametrization to help scale up neural networks.
  • Performance Monitoring: Uses CUDA event-based timing for precise GPU performance measurement.

Related

  • Project
    NVlabs/LongLive

    LongLive is NVIDIA’s open‑source suite for very long video generation and long‑context robot‑action modeling. It contains four self‑contained projects—Long‑WAM (world‑action models with streaming visual memory), LongLive‑Plug (once‑for‑all LoRA distillation), LongLive 2.0 (NVFP4‑based high‑performance training/inference infrastructure), and LongLive 1.0 (real‑time interactive text‑to‑video). The repo provides code, training recipes, inference pipelines, and pretrained checkpoints (hosted on HuggingFace) for each component, all under Apache‑2.0.

  • Project
    mll-lab-nu/RAGEN

    RAGEN is a reinforcement learning framework for training reasoning LLM agents over multiple interaction turns, built on the StarPO algorithm. It includes 10 built-in environments, supports PPO and GRPO optimization, and provides diagnostics for failure modes like the Echo Trap and reasoning collapse. RAGEN-2 adds SNR-Adaptive Filtering and mutual-information-based metrics to detect template collapse.

  • Project
    huggingface/setfit

    An efficient, prompt-free framework for few-shot fine-tuning of Sentence Transformers, enabling high-accuracy text classification with very little labeled data.

  • Project
    PythonOT/POT

    A comprehensive Python library for solving Optimal Transport problems, providing differentiable solvers for machine learning tasks like domain adaptation, data alignment, and generative modeling across multiple deep learning frameworks.