zhaochenyang20/Awesome-ML-SYS-Tutorial

What this is

Awesome-ML-SYS-Tutorial is an ongoing series of blog-length engineering notes — written by one person over about a year — about ML systems (ML SYS) and AI infrastructure (AI Infra): the software that makes large language models (LLMs) and multimodal models actually run fast, reliably, and correctly.

The author started these notes in August 2024 while learning to use SGLang (an inference engine for LLMs). Since then the collection has grown from a handful of articles and ~30–50 GitHub stars to 4,500+ stars. Content is largely about reinforcement-learning (RL) infrastructure, online/offline inference systems, and AI Infra fundamentals.

Why the author writes this

The core motivation is a conviction that the conclusions in research papers about reinforcement learning are only as trustworthy as the infrastructure they run on. The author has worked in RL infra development for over a year and observed that both open-source and company-internal RL infrastructure often has serious flaws. They question whether high-level research findings derived from such flawed systems can be considered correct — and they want to help build a correct foundation so that the conclusions drawn from it are trustworthy. This is also why they're involved in the SGLang RL community (they joined RadixArk full-time, continuing RL-infra research, and encourage readers to join the SGLang open-source community).

Who wrote it

The author (zhaochenyang20) studied AI Infra under advisors Professor Quanquan Gu, Dr. Ying Sheng, and Dr. Linmin Zheng. They began a Ph.D. path at UCLA but did not complete it for personal reasons; they now work full-time on RL infrastructure at RadixArk. They describe themselves as having become "a minor technical influencer" thanks to the project.

What's inside

The README organizes the notes into several themed sections. Each article typically has an English version, a Chinese version, and often a link to a Zhihu (Chinese Q&A platform) post. Many entries are deep source-code walk-throughs, optimization reports, or paper analyses. Here's what the README covers:

1. Omni Model Inference Notes

Articles about "omni" models — multimodal models that handle text, audio, images, etc. Topics include:

A deep dive into how Qwen3-Omni (a model) does inference, covering components like codecs and "Dual AR".
What actually runs out of memory when the SGLang inference engine hits an OOM (out-of-memory) error.
Why SGLang Omni was redesigned for multi-stage generative models, based on their decoding characteristics.
How text-to-speech (TTS) inference was made 1.9–3.4× faster in SGLang Omni via engineering techniques like caching and streaming.
Refactoring six architecturally different TTS backends around shared contracts (net removal of ~2,840 non-test lines).
How CPU resources affect speech model serving — including a fascinating case where a 4× performance swing in CI turned out to be host CPU contention, not code changes.
Batch-size selection trade-offs in SGLang Omni.

2. RLHF System Development Notes

This is the largest section, covering RLHF (Reinforcement Learning from Human Feedback) — the training technique that aligns models with human preferences. It's split into several frameworks and themes:

slime Framework — a specific RL framework the author works on. Articles cover multi-turn RL for LLMs and vision-language models (VLMs), INT4 quantization for RL training, adding an FSDP2 training backend (more flexible than Megatron for model architectures like Qwen3-Next and gpt-oss), FP8 precision for stable MoE RL, speculative decoding (using a small "draft" model to speed up sampling), source-code walk-throughs, and a chunked parallel computation of GAE (an RL advantage-estimation algorithm) that gives ~100–300× speedup.
AReal Framework — a source-code walk-through.
System Design and Optimization — deep dives into DeepSeek's Mixture-of-Experts architecture with FSDP, weight update mechanisms (half a year of work summarized), FSDP training backend principles, Megatron's basic features, integration of SGLang into OpenRLHF (described as "very painful" with a remaining nccl hang error), and DuoAttention for long-context inference implemented with CUDA's CuTeDSL.
verl Framework — source-code analyses (initialization, rollout, make experience, AgentLoop), parameter reference, memory-leak analysis via torch memory snapshots, latency optimization for weight updates, multi-turn training and tool calling, integration of Search-R1, and an analysis of the HybridFlow paper that underlies verl.
OpenRLHF Framework — PPO principles with source-code walk-throughs (including an illustrated series by another author, "Ms. Mengyuan"), and a brief analysis of post-training system computational flow.
Algorithms and Theory — notes on the TEMPO approach for competitive math (IMO) with long-horizon agentic RL, Kimi K1.5's long-context RL, rule-based rewards, SWE-Bench as a benchmark (how to build good evaluation tasks), and a survey of mainstream alignment algorithms plus NeMo-Aligner.

3. SGLang Learning Notes

Focused on the SGLang inference engine itself:

Diffusion Learning Notes — how SGLang supports diffusion-style LLMs (e.g., LLaDA 2.0), and a code walk-through of the diffusion handling pipeline.
Core Architecture and Optimization — begins an article on KV cache scheduling ("From KV Cache to Zero Overhead Scheduling"), though the README text is truncated at this point, so this section's full contents aren't shown.

Tags & status of articles

Some entries are marked [Pending Review] (not yet fully vetted) and one is marked [Not finished]. The README also notes that the author is a strong advocate for open-source collaboration — they explicitly invite interested readers to join the SGLang community.

One caveat: the README itself is cut off mid-sentence at the end (during the SGLang Learning Notes section), so this summary covers everything visible in the provided README.