hogeheer499-commits/strix-halo-guide

Evidence-backed AMD Strix Halo local-AI setup and benchmarks: Qwen3.8, Ollama, llama.cpp, Vulkan/ROCm, large GGUFs, and cross-OEM results.

What it solves

This project provides a comprehensive, independent technical guide for setting up and optimizing local Large Language Models (LLMs) on AMD Strix Halo / Ryzen AI MAX+ 395 systems. It addresses the difficulty of configuring BIOS, OS, and drivers to leverage the Radeon 8060S GPU and unified memory (96GB/128GB) for high-performance AI inference.

How it works

The guide offers a step-by-step setup process involving Ubuntu 24.04, specific kernel parameters for memory management, and the installation of the kisak Mesa PPA for updated drivers. It focuses on the Vulkan/RADV backend for tools like Ollama and llama.cpp to ensure the iGPU is utilized rather than falling back to the CPU. It also includes a curated set of "Best Known Profiles" and benchmarks to help users choose the right model and backend based on their specific needs (e.g., speed vs. quality).

Who it’s for

  • Owners or evaluators of AMD Strix Halo / Ryzen AI MAX systems.
  • Developers wanting to run local AI on AMD hardware.
  • Users seeking independent, evidence-based performance data for local LLM deployment.

Highlights

  • Automated Setup: Includes a setup.sh script for rapid deployment of the Linux-side Vulkan/RADV + Ollama stack.
  • Evidence-Based Benchmarks: Maps every performance claim to raw logs and CSVs, including tests for models like Qwen and Gemma.
  • Capacity Proof: Demonstrates the ability to run very large models, such as the DeepSeek V4 Flash 284B GGUF.
  • Cross-OEM Validation: Includes community results from 13 different systems to ensure portability across different hardware vendors.
  • Coded Workflow: Provides a verified route for fine-tuning and exporting models using the ROCm 7.2 Unsloth route.

Related

  • Dispatch
    AMD Strix Halo RDMA Cluster Setup Guide

    This guide explains how to configure a two-node AMD Strix Halo cluster using Intel E810 RoCE v2 NICs to enable low-latency distributed vLLM inference via Tensor Parallelism.

  • Dispatch
    llama.cpp and llama.app: Local LLM Inference and Ecosystem

    llama.cpp and its official home llama.app provide high-performance, hardware-agnostic local LLM inference, now featuring a simplified 'llama serve' command and integration with local coding agents like Pi.

  • Dispatch
    vLLM speculative decoding on AMD GPUs: performance and methods

    vLLM adds speculative decoding to AMD Instinct MI300X/MI355X GPUs, letting a lightweight draft model propose multiple tokens that the target model verifies in a single pass, which can double or more the output-token throughput depending on the draft method, model family, and proposal length.

  • Project
    jmaczan/tiny-vllm

    tiny‑vllm is an open‑source C++/CUDA LLM inference engine (with optional AMD‑HIP support) that loads a Llama 3.2 1B model from Safetensors and implements the full forward pass—including custom kernels for embeddings, RMSNorm, RoPE, grouped‑query attention, Flash‑style softmax, and paged KV‑cache. It doubles as a step‑by‑step tutorial covering every component of transformer inference, aimed at learners and educators.

  • Project
    xLLM-AI/xllm

    xLLM is an open‑source C++ inference framework that delivers high‑throughput, low‑latency LLM serving on Chinese AI accelerators (Ascend, Cambricon, Moore Threads, etc.). It separates a brpc‑based service layer from a hardware‑optimized engine, includes hybrid KV‑cache management via Mooncake, and provides ready‑to‑run deployment scripts for models such as GLM‑5.3‑Flash and MiniMax‑M3. The project is Apache‑2.0 licensed and backed by JD.com and several Chinese university labs.