pytorch/FBGEMM

FB (Facebook) + GEMM (General Matrix-Matrix Multiplication) - https://code.fb.com/ml-applications/fbgemm/

What it solves

FBGEMM provides highly-optimized kernels for deep learning applications, focusing on making server-side inference faster and more efficient by using low-precision arithmetic. It addresses the challenges of bandwidth-bound operations and accuracy loss typically associated with low-precision matrix multiplication and convolutions.

How it works

The project is divided into three specialized packages:

  • FBGEMM: A library for x86 machines that provides high-performance, low-precision general matrix multiplication (GEMM) and convolution, utilizing techniques like row-wise and outlier-aware quantization to maintain accuracy.
  • FBGEMM_GPU: A collection of PyTorch GPU operator libraries designed for training and inference, with a specific focus on recommendation systems.
  • FBGEMM_GPU GenAI: A collection of GPU operator libraries tailored for generative AI, providing features like FP8 row-wise quantization and collective communications.

Who it’s for

Developers and researchers working on deep learning inference engines, specifically those using PyTorch on x86 servers or GPUs, who need to maximize performance and optimize low-precision operations.

Highlights

  • Low-Precision Optimization: Specialized kernels for high-performance matrix multiplication and convolution.
  • PyTorch Integration: Serves as a backend for PyTorch quantized operators on x86 machines.
  • Quantization Techniques: Supports row-wise and outlier-aware quantization to minimize accuracy loss.
  • GenAI Support: Dedicated GPU operators for generative AI, including FP8 quantization.
  • Broad Hardware Support: Optimized for x86 servers and GPUs, with community ports for Huawei Ascend NPU.

Related

  • Project
    flashinfer-ai/flashinfer

    A high-performance GPU kernel library and generator for LLM inference that optimizes attention, GEMM, and MoE operations across diverse NVIDIA GPU architectures.

  • Project
    deepseek-ai/DeepGEMM

    A high-performance tensor core kernel library for NVIDIA GPUs that provides optimized GEMM, MoE, and MQA kernels to accelerate LLM training and inference.

  • Project
    ztxz16/fastllm

    A high-performance C++ LLM inference library that enables running massive models like DeepSeek R1 on limited hardware through hybrid GPU-CPU inference and PyTorch-free operators.

  • Project
    ROCm/aiter

    AITER is AMD's high-performance AI operator library providing optimized GPU kernels for inference and training workloads on ROCm-compatible GPUs.

  • Project
    cfregly/ai-performance-engineering

    A comprehensive collection of code and resources for optimizing AI systems, focusing on GPU tuning, distributed training, and high-throughput inference scaling.