ROCm/FastFlowLM

Run LLMs on AMD Ryzen™ AI NPUs in minutes; purpose-built and deeply optimized for the AMD NPUs.

What it solves

FastFlowLM (FLM) allows users to run large language models (LLMs) and vision-language models (VLMs) directly on AMD Ryzen™ AI NPUs. This removes the dependency on a GPU or CPU for inference, resulting in significantly lower power consumption and higher efficiency.

How it works

FLM provides a lightweight (17 MB) NPU-first runtime and a single-command CLI. It uses optimized binary kernels to execute models on XDNA2 NPUs (found in Ryzen™ AI Series chips). It supports a variety of model types, including Vision, Audio, Embedding, and Mixture-of-Experts (MoE), and can be deployed as a CLI tool or a local REST/OpenAI-compatible API server.

Who it’s for

Developers and users with AMD Ryzen™ AI NPUs who want to run local AI models privately and efficiently without needing a GPU.

Highlights

  • NPU-Native Execution: Runs fully on AMD Ryzen™ AI NPUs with no GPU or CPU load.
  • High Efficiency: Over 10x more power-efficient than traditional methods.
  • Broad Model Support: Supports LLMs, VLMs, and MoE models, including Vision and Audio capabilities.
  • Long Context: Supports context windows up to 256k tokens.
  • Developer Friendly: Features a simple CLI and an OpenAI-compatible API server.

Related

  • Project
    NotPunchnox/rkllama

    An LLM server and client optimized for Rockchip RK3588/3576 platforms that enables AI models to run on the NPU for accelerated local inference.

  • Project
    amd/RyzenAI-SW

    A suite of tools and runtime libraries for optimizing and deploying AI inference on AMD Ryzen AI-based PCs to maximize hardware acceleration.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    ztxz16/fastllm

    FastLLM is a C++‑based, PyTorch‑free inference engine for large language models. It supports dense and MoE models, hybrid GPU/CPU/NUMA/disk deployment, multi‑GPU tensor parallelism, speculative decoding (MTP/DSpark/DFlash2), many quantization formats, and offers a Python CLI, web UI, and OpenAI‑compatible API server. Install via `pip install ftllm`, run a model with `ftllm run`, or launch a service with `ftllm server`. Extensive docs cover model‑specific guides, performance tuning, and community support.

  • Project
    joaopauloschuler/neural-api

    A native Pascal deep-learning library that enables running LLMs, transformers, and other AI models as single native binaries without Python or CUDA.