dorjeduck/llm.mojo

port of Andrjey Karpathy's llm.c to Mojo

What it solves

It demonstrates that the Mojo programming language can implement low-level, C-style AI training code—utilizing raw pointers and manual memory management—while matching or exceeding the performance of C.

How it works

This project is a direct port of an earlier version of Andrej Karpathy's llm.c (specifically the train_gpt2.c implementation). It implements the GPT-2 training process in Mojo, allowing users to train a model on datasets like TinyShakespeare.

Who it’s for

Developers and researchers interested in the performance characteristics of Mojo compared to C for low-level machine learning implementations.

Highlights

  • High Performance: Benchmarks on an M2 MacBook Pro show the Mojo implementation achieving higher throughput (150 tok/s) than the OpenMP-enabled C version (128 tok/s).
  • Low-Level Control: Implements raw pointers and manual memory management to mirror the original C project.
  • Validation: Includes a ported version of test_gpt2.c to ensure the functionality of the port is correct.

Related

  • Dispatch
    Mojo 1.0 Beta: Bridging the Gap Between Python Productivity and C++ Performance

    Mojo enters beta as a compiled, statically-typed language designed for AI and systems programming, promising the syntax of Python with the performance of C++.

  • Dispatch
    Mojo Language Open Source Release

    Modular has open-sourced the Mojo compiler and toolchain under the Apache 2.0 license, enabling developers to build and customize the systems language for AI accelerators.

  • Project
    vixhal-baraiya/microgpt-c

    A minimal, dependency-free GPT implementation in pure C that allows for character-level training and inference without external ML frameworks.

  • Project
    go-skynet/go-llama.cpp

    Go bindings for llama.cpp that enable high-performance local LLM inference within Go applications using the GGUF format.

  • Project
    modular/mojo-gpu-puzzles

    An interactive, open‑source tutorial that teaches GPU programming in Mojo through hands‑on puzzles, complete with a `pixi`‑based environment, test harnesses, and community support.