naklecha/simple-llm

~1300 line, minimal, extensible LLM inference engine built from scratch.

What it solves

SimpleLLM is a lightweight, high-performance inference engine designed specifically to run OpenAI's gpt-oss-20b and gpt-oss-120b models on a single NVIDIA H100 GPU. It aims to provide faster decoding speeds than vLLM for specific workloads while maintaining the accuracy of the original models.

How it works

The engine is implemented in approximately 1,351 lines of Python, utilizing Triton for GPU kernels. It loads official MXFP4 checkpoints directly without re-quantization. Key performance optimizations include:

  • Custom Kernels: A single decode-attention kernel that handles sinks, sliding windows, GQA, and KV append, and a fused RoPE kernel.
  • Memory Management: A paged KV cache for full-attention layers and a ring buffer for sliding-window layers to optimize context handling.
  • Execution Efficiency: Use of CUDA graphs for active slots and greedy argmax/next-step inputs computed directly inside the CUDA graph.
  • MoE Optimization: MXFP4 Mixture-of-Experts (MoE) implementation on tensor cores and optimized routing that writes tile tables per expert.

Who it’s for

Developers and researchers who need to run OpenAI's gpt-oss models on a single H100 and want maximum decoding throughput, particularly for small to medium batch sizes or long context lengths.

Highlights

  • Performance: Decodes 11-15% faster than vLLM 0.31.0 at batch 1 and 64.
  • Efficiency: Extremely compact codebase (under 1,400 lines of Python).
  • Accuracy: Matches OpenAI's published GPQA-Diamond and AIME 2025 scores.
  • Hardware Optimized: Specifically tuned for the NVIDIA H100 (80 GB) using MXFP4 weights.

Related

  • Project
    Arm-Examples/ML-examples

    A collection of machine learning examples and tutorials for deploying and optimizing AI models on Arm Ethos NPUs and Cortex-based platforms.

  • Project
    volcano-sh/kthena

    A lightweight, Kubernetes-native LLM inference platform that provides modular control and data planes for scalable, cost-efficient model serving.

  • Project
    snowflakedb/ArcticInference

    An open-source vLLM plugin that optimizes LLM and embedding inference to increase throughput and reduce latency using advanced parallelism and speculative decoding.

  • Project
    tonyd2wild/DeepSeek-v4-Flash-Vision-Exp-DSpark-1M-NVFP4-KV-2x-DGX-Spark

    A deployment recipe for serving DeepSeek-V4-Flash-Vision-Exp on DGX Spark clusters using vLLM, featuring 1M context support and DSpark speculative decoding.