EleutherAI/gpt-neox

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

What it solves

GPT-NeoX is designed for the training of large-scale autoregressive language models with billions of parameters from scratch. It provides a centralized framework to gather and implement training techniques that accelerate research into large-scale model development.

How it works

Built upon NVIDIA's Megatron Language Model and augmented with DeepSpeed techniques, the library implements distributed training using ZeRO and 3D parallelism. It supports a wide range of hardware (NVIDIA and AMD GPUs) and orchestration tools like Slurm, MPI, and the IBM Job Step Manager. The framework incorporates architectural innovations such as rotary and alibi positional embeddings, parallel feedforward attention layers, and flash attention to optimize performance.

Who it’s for

Researchers and organizations in academia, industry, and government who need to train massive language models from scratch. It is specifically not intended for generic inference needs, for which the authors recommend using the Hugging Face transformers library.

Highlights

  • Broad Hardware Support: Compatible with NVIDIA and AMD GPUs across various cloud and supercomputing environments (e.g., AWS, CoreWeave, LUMI).
  • Architectural Flexibility: Includes predefined configurations for popular architectures like LLaMA 1 & 2, PaLM, Falcon, and Pythia.
  • Advanced Optimizations: Supports Flash Attention and NVIDIA's Transformer Engine for significant speed-ups on Ampere and Hopper GPUs.
  • Ecosystem Integration: Connects with Hugging Face tokenizers and transformers, and integrates with monitoring tools like WandB, Comet, and TensorBoard.

Related

  • Project
    NVIDIA-NeMo/Megatron-Bridge

    NVIDIA‑NeMo Megatron‑Bridge is a PyTorch library that links Hugging Face models with NVIDIA’s Megatron‑Core training engine. It enables bidirectional checkpoint conversion, large‑scale pre‑training and fine‑tuning (including LoRA), and provides verified recipes for many recent LLM/VLM/MoE models. The recommended workflow uses NVIDIA’s NeMo Docker container and a simple Python API (`AutoBridge`).

  • Dispatch
    Training Language Models with Megatron-LM

    Hugging Face provides a guide on using NVIDIA's Megatron-LM framework to efficiently pre-train large language models on GPUs, including integration with the Transformers library.

  • Dispatch
    Hugging Face Train on DGX Cloud

    Hugging Face launched Train on DGX Cloud, a no-code service for Enterprise Hub organizations to fine-tune open models using NVIDIA H100 and L40S GPUs.

  • Dispatch
    🤗 PEFT library release enables parameter-efficient fine-tuning of billion‑scale models

    Hugging Face announced the 🤗 PEFT library, which lets users fine‑tune billion‑parameter models on low‑resource hardware by training only a tiny fraction of parameters.

  • Project
    AI4Finance-Foundation/FinGPT

    FinGPT is an open‑source suite of financial large language models fine‑tuned with LoRA/QLoRA for tasks like sentiment analysis, relation extraction, headline classification, NER, QA, and stock‑price forecasting. It offers multi‑task and single‑task checkpoints, public instruction‑tuning datasets, a benchmark suite, and demo interfaces, all under an Apache‑2.0 license.