changyeyu/LLM-RL-Visualized

What is this project?

LLM-RL-Visualized is a collection of 100+ original architecture diagrams that explain how large language models (LLMs) and reinforcement learning (RL) work. The diagrams cover topics like:

• LLM / VLM (vision-language model) principles
• Training algorithms: RL, RLHF, GRPO, DPO, SFT, CoT distillation
• Effect optimization and RAG (retrieval-augmented generation)

It's a visual learning resource — the author draws diagrams to make complex AI concepts easier to understand.

What's inside?

The repository is organized into 11 parts, each covering a different area:

1. Technical overview maps — big-picture diagrams of LLM and RL algorithms
2. LLM basics — model structure, input/output layers, decoding, multimodal models, training pipeline
3. SFT (supervised fine-tuning) — fine-tuning techniques, LoRA, Prefix-Tuning, loss functions, data handling
4. DPO (direct preference optimization) — how DPO works, how it compares to RLHF, training diagrams
5. Training-free optimization — CoT (chain of thought), search strategies, sampling methods, RAG, function calling
6. RL fundamentals — Markov processes, value functions, DQN, policy gradient, multi-agent RL, imitation learning
7. Policy optimization architectures — Actor-Critic, PPO, TRPO, GAE, DDPG, GRPO
8. RLHF & RLAIF — reward models, PPO-based RLHF, rejection sampling, constitutional AI
9. Reasoning optimization — knowledge distillation, MCTS, best-of-N sampling, majority vote
10. LLM extensions — quantization, position encodings, normalization, pruning, benchmarks
11. Appendix — text explanations, references, citation formats

Key features

• High-resolution images: Click any diagram to see a large version, or browse the .svg vector files (infinitely zoomable, selectable text)
• Bilingual: Available in Chinese and English (two README versions)
• Bonus PDFs: Includes a large "Reinforcement Learning Algorithm Map" PDF and a "Policy Gradient" derivation PDF
• Companion book: The author also wrote a book called 《大模型算法：强化学习、微调与对齐》 (Large Model Algorithms: RL, Fine-tuning, and Alignment), which provides more detailed explanations of the diagrams
• Actively maintained: The author states the repository is continuously corrected and expanded

Who is this for?

Anyone who wants to visually understand how modern AI models are built and trained — from the internal structure of an LLM to the math behind reinforcement learning algorithms like PPO and GRPO. The diagrams are designed to make these dense topics more approachable.

How to use it

The README is essentially a table of contents — each entry links to a diagram (PNG or SVG) with a short explanation. You can browse by topic, click through to see the full-size image, and follow the links to the PDF files for deeper dives.

Note: The README mentions the project is featured on HelloGitHub, a platform that recommends open-source projects.