karpathy/micrograd

A tiny scalar-valued autograd engine and a neural net library on top of it with PyTorch-like API

What it solves

micrograd is a tiny, educational autograd engine that implements backpropagation (reverse-mode autodiff) over a dynamically built Directed Acyclic Graph (DAG). It allows users to build and train small neural networks using a PyTorch-like API, making the internal mechanics of gradient calculation and network training fundamentally understandable.

How it works

The engine operates exclusively over scalar values, breaking down every operation (like addition and multiplication) into individual steps. It builds a DAG of these operations, which then allows the engine to calculate gradients via a backward pass. On top of this engine, a small neural network library is provided to build Multi-Layer Perceptrons (MLPs) and binary classifiers.

Who it’s for

Developers and students who want to understand the core mathematical foundations of deep learning and how automatic differentiation works without the complexity of a large-scale production framework.

Highlights

  • Tiny Footprint: The autograd engine is roughly 100 lines of code, and the neural network library is about 50 lines.
  • PyTorch-like API: Provides a familiar interface for those acquainted with modern deep learning libraries.
  • Visualization: Includes tools to produce Graphviz visualizations of the computation graph, showing both data and gradients.
  • Educational Focus: Designed specifically for educational purposes to demonstrate binary classification and the basic building blocks of neural networks.

Related

  • Project
    tinygrad/tinygrad

    A lightweight, hackable end-to-end deep learning stack that provides a PyTorch-like API with a transparent compiler and IR for efficient hardware acceleration.

  • Project
    HIPS/autograd

    A library that automatically differentiates native Python and NumPy code to enable efficient gradient-based optimization.

  • Project
    pytorch/pytorch

    A Python-based machine learning framework that provides GPU-accelerated tensor computation and a dynamic autograd system for building deep neural networks.

  • Project
    f-dangel/backpack

    A PyTorch extension that efficiently computes quantities beyond the standard gradient, such as per-sample gradients and gradient variance, by reusing information during the backward pass.

  • Project
    AgwaB/pi-workflow

    A workflow orchestration system for Pi that enables the execution of repeatable, multi-step AI agent workflows for tasks like research and code review.