YeonwooSung/llm-architecture-from-scratch

What it solves

This project provides clean, educational implementations of popular Large Language Model (LLM) architectures from scratch. It is designed to help developers and researchers gain a deep, practical understanding of how modern AI models are structured, from the underlying attention mechanisms to complex components like Mixture of Experts (MoE).

How it works

The repository contains modular codebases for specific, well-known models (such as Llama 3, DeepSeek v3, and Gemma 3). Each folder implements the model's specific architectural choices—such as normalization layers (LayerNorm vs. RMSNorm), positional encodings (RoPE), and attention variants (Grouped-Query Attention or Sliding Window)—allowing users to inspect the code to see how these components interact in a functional model.

Who it’s for

This project is for students, machine learning engineers, and researchers who want to move beyond high-level API usage and understand the low-level implementation details of state-of-the-art LLM architectures.

Highlights

• Includes implementations for a wide range of model sizes, from 3B to 671B parameters.
• Covers diverse architectural features including Mixture of Experts (MoE) and multimodal capabilities.
• Implements modern attention mechanisms like Grouped-Query Attention (GQA) and sliding window attention.
• Provides a clear comparison of normalization and positional encoding techniques across different model families.