milanm/AutoGrad-Engine

What it solves

MicroGPT.cs is an educational tool that makes the inner workings of transformer-based GPT models accessible. It shows a complete, working implementation in pure C# with no external libraries, so you can see exactly how tokenization, embeddings, attention, MLPs, backpropagation, and generation fit together. Instead of black-box frameworks, you get a transparent, commented codebase that demystifies the algorithm behind ChatGPT.

How it works

The project is a faithful port of Andrej Karpathy's microgpt.py. It consists of four core files:

• Value.cs — an autograd engine that wraps scalars and automatically tracks gradients.
• Tokenizer.cs — character-level encoding and decoding (BOS/EOS tokens).
• NeuralOps.cs — stateless building blocks: Linear, Softmax, RMSNorm.
• Program.cs — the GPT model, training loop, and generation logic.

Training uses a tiny dataset of human names (32,033 entries). The model is a standard transformer with configurable hyperparameters (e.g., embedding size, number of layers, attention heads). During training, cross-entropy loss is minimized with the Adam optimizer. After training, new names are generated autoregressively: start with BOS, sample the next character from softmax probabilities, feed it back, and repeat until EOS or max length.

The README also includes a detailed developer's guide that walks through each step, from tokenization to backpropagation, with analogies and diagrams.

Who it’s for

It's aimed at developers and machine learning enthusiasts who want to truly understand GPT. The code is heavily commented and includes a prerequisites guide for beginners, plus comparisons to production-scale models (like GPT-4) to illustrate the difference. If you can write a for-loop, you can follow along. It is intentionally not production code — it processes one number at a time rather than parallel tensors, but every conceptual piece is present.

Highlights

• Pure C# with zero dependencies: no PyTorch, TensorFlow, or NuGet packages.
• ~600 lines of code across 4 files, with extensive comments.
• Includes a full autograd engine that tracks gradients automatically, verified with numerical gradient checking (similar to PyTorch's gradcheck).
• Modern design choices (RMSNorm instead of LayerNorm, no biases, squared ReLU, pre-norm) closer to LLaMA than GPT-2.
• CLI arguments let you tweak model size, training steps, learning rate, and more.
• Clear comparison table against GPT-4 to show the scale difference (parameters, context, training data).