NanoEuler: A GPT-2 Scale LLM Built from Scratch in C and CUDA

NanoEuler implements a GPT-2-style LLM without high-level ML libraries

NanoEuler is a research and educational project that implements a GPT-2-class language model entirely from scratch using C and CUDA. The project avoids all machine learning libraries, including PyTorch and autograd, requiring the author to hand-write both the forward and backward passes for the entire training pipeline. It is designed to demonstrate a complete, understandable training pipeline—from a byte-level BPE tokenizer to supervised fine-tuning (SFT)—rather than to serve as a production-ready assistant.

Technical Architecture and Model Specifications

NanoEuler utilizes a decoder-only transformer architecture incorporating several modern LLM components to optimize performance and representation learning:

  • RMSNorm: Used as pre-normalization without bias.
  • Rotary Position Embeddings (RoPE): Applied to queries and keys to handle positional information.
  • SwiGLU Feed-Forward: Implemented as down(silu(gate(x)) * up(x)).
  • Grouped-Query Attention (GQA): Query heads share a smaller set of key/value heads to reduce memory overhead.
  • Multi-Token Prediction (MTP): The model uses K output heads to predict the next K tokens, which improves learned representations and supports speculative decoding.
  • Byte-level BPE Tokenizer: A hand-written tokenizer with GPT-2-style pretokenization that prevents spaces from being wasted as standalone tokens.

Model Configurations

Configuration Dimension Q/KV Heads Layers Context Vocab Parameters
Small (CPU) 128 4 / 2 4 128 512 ~1.05M
GPU Pipeline 768 12 / 4 16 512 4096 ~116M

CUDA Engine and Performance Optimizations

The GPU implementation (cuda/nanoeuler_cuda.cu) provides a full port of the training and inference pipeline. To achieve efficiency on a single RTX 4070, the project employs several specific optimizations:

  • cuBLAS Integration: Matrix multiplications are delegated to cuBLAS using TF32 tensor cores.
  • FlashAttention: A hand-written, tiled implementation with online softmax that avoids storing the full $T \times T$ matrix in memory. This optimization reportedly makes the training step approximately 3x faster.
  • GPU Validation: Every kernel is validated against a CPU reference, and the entire model undergoes a GPU-to-CPU gradient check to ensure accuracy within $1e-6$.

Verified Backpropagation and Training Pipeline

Because hand-written backpropagation is prone to subtle errors, NanoEuler verifies every analytic gradient against a central finite difference. This check is performed in double precision to prevent floating-point cancellation from masking errors. The verification covers all parameters, including the backward passes for RoPE, SwiGLU, GQA, and MTP.

The Pretraining and SFT Workflow

The project demonstrates a two-stage pipeline to move from a base model to a chat-capable model:

  1. Pretraining: The ~116M parameter model is pretrained on a mix of Project Gutenberg classics and a slice of the FineWeb-Edu dataset (accessed via DuckDB CLI to avoid Python dependencies).
  2. Supervised Fine-Tuning (SFT): The pretrained base is fine-tuned using the Alpaca dataset. The loss is masked to response tokens only, ensuring the model learns to generate the response rather than the prompt.

Conceptual Basis: The "Euler" Connection

The project derives its name from Leonhard Euler and the forward-Euler method of numerical integration. The author posits that a residual block ($x = x + f(x)$) can be viewed as a single step of numerical integration for an ordinary differential equation (ODE) where $\Delta t = 1$. In this view, the depth of a deep residual network corresponds to integration time, and each layer advances the hidden state by one Euler step.

Community Insights and Critiques

While the project has been praised for its engineering effort, some community members on Hacker News have raised questions regarding the implementation and documentation:

"Mentioning neural ODE doesn't make sense here... Basically any implementation of transformer uses residuals, but you're not really training a neural ODE here."

Other users questioned the coding style and the extent to which LLMs were used to generate the README and the source code, noting specific comments in the CUDA source that suggested certain sections were untested.

Sources

Related