vllm-project/humming

Humming is a high-performance, lightweight, and highly flexible JIT (Just-In-Time) compiled GEMM kernel library specifically designed for quantized inference.

What it solves

Humming addresses the need for high-performance, flexible matrix multiplication (GEMM) kernels specifically optimized for quantized neural network inference. It allows developers to run models using various low-bit precision formats without sacrificing speed or compatibility.

How it works

It uses a Just-In-Time (JIT) compilation approach to generate optimized kernels on the fly. By supporting a wide range of weight types (from INT4 to FP8) and activation types, it allows users to define a layer configuration, load weights, and perform high-speed quantized calculations directly on NVIDIA GPUs.

Who it’s for

This library is intended for developers and researchers building or optimizing AI inference engines who need to implement custom quantization strategies or support diverse low-bit precision formats on modern NVIDIA hardware.

Highlights

  • Supports a vast array of precision formats including FP16, BF16, FP8, FP4, INT8, and INT4.
  • Highly compatible with NVIDIA GPUs from the Turing architecture (SM75) and newer.
  • Extremely lightweight with a package size of just over 100KB and minimal dependencies.
  • Optimized for both standard Dense GEMM and Mixture-of-Experts (MoE) architectures.
  • Provides state-of-the-art throughput and efficiency for quantized workloads.

Related

  • Project
    PipeNetwork/kimi-k3-mlx

    An MLX port of Moonshot's Kimi-K3 multimodal MoE model, featuring a streaming converter and REAP pruning to enable execution on Apple Silicon.

  • Project
    HELPMEEADICE/TE-Speed-MiniMaxH3-OSS

    A ComfyUI custom node that accelerates MiniMax H3 model inference by up to 45% using a residual caching mechanism for DiT blocks.

  • Project
    Headmaster218/MoE4All

    MoE4All is a high-performance inference engine that enables running large Mixture-of-Experts models on consumer hardware by intelligently offloading weights between SSD, RAM, and GPU VRAM.

  • Project
    Dicklesworthstone/franken_ocr

    franken_ocr is a pure-Rust, CPU-only engine that runs specialized vision-language models for document parsing, chart extraction, and music transcription without requiring GPUs or Python.