anuj0456/OpenArch

OpenArch – Readable PyTorch Implementations of Modern LLM Architectures

What it is – A collection of hand‑written PyTorch model.py files, each implementing a single large‑language‑model (LLM) architecture from Sebastian Raschka’s LLM Architecture Gallery. The code is deliberately minimal: one readable file per model, no sharding, no speed‑optimisation tricks, just the core layers needed for a forward pass.

Why it exists – Production libraries such as Hugging Face transformers are dense and optimized for performance, which makes it hard to see why a model is built the way it is. OpenArch aims to be a learning resource where the architectural choices (attention type, normalization, positional encoding, MoE routing, etc.) are explicit and easy to compare side‑by‑side.

What’s implemented – The README lists dozens of text‑only models (GPT‑2 XL, Llama 2/3/4, OLMo 2, DeepSeek R1, Gemma 3, Mistral 3, Qwen 3 variants, Kimi K2, GLM 4.5, GPT‑OSS, Grok‑2.5, etc.) and a few multimodal or image models. Each entry shows:

• Model size (e.g., 7 B, 400 B)
• Normalization (RMS Norm, QK‑Norm, etc.)
• Positional encoding (RoPE, absolute, etc.)
• Attention variant (multi‑head, grouped‑query, sliding‑window, latent, gated)
• Whether it uses a mixture‑of‑experts (MoE)

✅ means the file can run a forward pass; 🚧 means the implementation is still in progress.

Repository layout

OpenArch/
├─ text/      ← one folder per text model (gpt2/, llama3/, …)
│   ├─ model.py
│   └─ README.md
├─ multimodal/ (e.g., pali‑gemma/)
├─ requirements.txt
└─ README.md (this file)

Each model folder contains a short README documenting the specific architectural choices and the sources consulted.

How to use it

1. Clone the repo and install the dependencies from requirements.txt (PyTorch and a few utilities).
2. Navigate to a model folder, e.g. cd text/llama3.
3. Run the provided model.py (or import it) to instantiate the model class and feed a tensor of token IDs. The code includes a simple forward‑pass test that can optionally load official weights and verify a few token outputs.

Contributing – The project welcomes contributors who want to:

• Add a missing architecture from the gallery.
• Write a README for an existing model.
• Add tests that compare against official weights.
• Improve docstrings or refactor shared utilities. Contributions should prioritize clarity over speed; the goal is education, not production.

License – Apache 2.0 for the repository overall; individual model files follow the licenses of the original models.

Bottom line – OpenArch is a genuine, open‑source learning‑oriented codebase for modern LLM architectures, useful for anyone who wants to read and understand the building blocks of today’s large language models without the overhead of a full‑blown library.