fastmachinelearning/hls4ml

Machine learning on FPGAs using HLS

What it solves

It enables ultra-low-latency machine learning inference on Field Programmable Gate Arrays (FPGAs). It bridges the gap between high-level machine learning frameworks and the specialized hardware requirements of FPGAs, allowing researchers to deploy models in environments where microseconds matter.

How it works

The package translates models from open-source machine learning libraries into High Level Synthesis (HLS) language. This HLS code is then synthesized into firmware that can be implemented on FPGA hardware. It supports multiple back-ends for synthesis, including Xilinx Vivado HLS, Vitis HLS, Intel HLS, and Catapult HLS.

Who it’s for

It is designed for scientists and engineers working in domains requiring extreme real-time performance, such as high-energy physics (e.g., CERN's Large Hadron Collider), quantum computing control systems, nuclear fusion feedback loops, satellite environmental monitoring, and biomedical signal processing.

Highlights

  • Ultra-low latency: Optimized specifically for high-speed inference on reconfigurable hardware.
  • Broad Framework Support: Converts traditional ML package models into HLS.
  • Hardware Flexibility: Compatible with various synthesis tools from Xilinx and Intel.
  • Diverse Model Support: Capable of handling CNNs, binary/ternary networks, and Spiking Neural Networks (SNNs).

Related

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.

  • Project
    Tencent/hpc-ops

    A production-grade operator library for LLM inference that provides highly optimized CUDA kernels for attention, MoE, and sampling to reduce serving latency.

  • Project
    sgl-project/sglang

    A high-performance serving framework for LLMs and multimodal models that provides low-latency, high-throughput inference across single GPUs and large distributed clusters.

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Project
    NVIDIA/TensorRT-LLM

    An optimization library for LLMs and visual generation models that uses specialized kernels and an efficient runtime to maximize inference throughput and minimize latency on NVIDIA GPUs.