AMD Strix Halo RDMA Cluster Setup Guide

Distributed inference for large language models (LLMs) typically requires high-speed interconnects to prevent communication bottlenecks. This guide details the configuration of a two-node AMD Strix Halo cluster linked via Intel E810 (RoCE v2), enabling distributed vLLM inference using Tensor Parallelism (TP) with ultra-low latency.

Architecture and Core Concepts

To achieve high performance in a distributed APU cluster, the system relies on the orchestration of four primary technologies:

• vLLM: The inference engine that splits models across multiple GPUs/APUs using Tensor Parallelism.
• Ray: The distributed computing framework that manages the control plane, orchestrating worker processes across nodes.
• RCCL (ROCm Collective Communication Library): The AMD equivalent of NVIDIA's NCCL. It handles the data plane, synchronizing tensor data between nodes. Because TP requires synchronization after every neural network layer, high-speed data exchange is critical.
• RoCE v2 (RDMA over Converged Ethernet): This protocol allows RCCL to write data directly from one node's memory to another, bypassing the CPU and OS kernel. This reduces latency from ~70-100µs (standard TCP/IP) to approximately 5µs.

Hardware Requirements

Building this cluster requires specific hardware to support 100GbE throughput and unified memory access:

• Compute Nodes: Two Framework Desktop Mainboards featuring AMD Ryzen AI MAX+ "Strix Halo" with 128GB of Unified Memory.
• Network Interface Cards (NICs): Intel Ethernet Controller E810-CQDA1 (or similar 100GbE QSFP28 cards).
• Interconnect: Direct Attach Copper (DAC) cables for a direct node-to-node connection, eliminating the need for a switch.
• PCIe Connectivity: Since the Framework motherboard PCIe slot is physically x4, a PCIe 4x to 16x riser is required to accommodate the x16 NICs. Performance remains consistent at ~50Gbps bandwidth and ~5µs latency.

Host Configuration (Fedora 43)

Both nodes must be configured on Fedora 43 (verified kernels 6.18.5-200.fc43.x86_64 and 6.18.6-200.fc43.x86_64).

Software and Firmware

Install the core RDMA userspace tools via DNF:

sudo dnf install rdma-core libibverbs-utils perftest

Verify that the Intel E810 firmware is at least version 4.91. Older firmware should be updated using the Intel Ethernet NVM Update Tool.

Network Setup

Assign static IPs (e.g., 192.168.100.1 and 192.168.100.2) on a /30 subnet. Set the MTU to 9000 (Jumbo Frames) to reduce CPU overhead. Verify the link state using rdma link, which should show the state as ACTIVE and LINK_UP.

BIOS and Kernel Tuning

To maximize the utility of the unified memory and RDMA performance, the following adjustments are required:

1. BIOS: Set iGPU Memory Allocation to the minimum (512MB). This allows the GTT (Graphics Translation Table) to dynamically allocate system memory as VRAM.
2. Kernel Parameters: Append the following to GRUB_CMDLINE_LINUX in /etc/default/grub:
   • iommu=pt: Enables IOMMU Pass-Through mode to reduce overhead for the NIC and iGPU.
   • pci=realloc: Reallocates PCI BARs to map large address spaces.
   • pcie_aspm=off: Disables PCIe Active State Power Management to prevent latency spikes.
   • amdgpu.gttsize=126976: Caps GPU GTT size to ~124GiB.
   • ttm.pages_limit=32505856: Matches the TTM limit to the GTT size (~124GiB).

Firewall

Trust the RDMA interface completely to avoid blocking the random high ports used by Ray and RCCL:

sudo firewall-cmd --permanent --zone=trusted --add-interface=enp194s0np0
sudo firewall-cmd --reload

Toolbox Installation and RDMA Verification

Because upstream ROCm packages currently lack gfx1151 (Strix Halo) support for RDMA, a custom-built librccl.so patch is required. This is provided via the kyuz0/vllm-therock-gfx1151 Docker image.

Run ./refresh_toolbox.sh on both nodes. This script pulls the patched image and configures the container to expose /dev/dri, /dev/kfd, /dev/infiniband, and sets --ulimit memlock=-1 for DMA memory pinning.

Verifying Latency

Run /opt/compare_eth_vs_rdma.sh from the head node. Expected results should show a drop from ~70ms (Ethernet) to ~5µs (RDMA).

Running the vLLM Cluster

Cluster management is handled via the start-vllm-cluster TUI utility:

1. Ray Cluster: Initialize the head node on Node 1 and the worker node on Node 2. Verify that the cluster sees two nodes and the combined GPU resources.
2. vLLM Serve: Select a model (e.g., Llama-3.1-8B) and set Tensor Parallelism to 2.
3. Critical Setting: Enable "Force Eager Mode". CUDA Graph capture can be unstable on distributed APU clusters, leading to deadlocks. Eager mode is safer and prevents these hangs.

Alternative: Thunderbolt Networking

For users without 100GbE NICs, nodes can be connected via a Thunderbolt 4 / USB4 cable. While this lacks the microprocessor-level latency of RDMA, it provides significantly higher bandwidth than standard Ethernet.

1. Configuration: Create a thunderbolt0 interface with static IPs (e.g., 192.168.2.1 and 192.168.2.2) and an MTU of 9000.
2. Execution: Use the start-vllm-cluster utility to set the IPs explicitly to the Thunderbolt interface addresses. The script will automatically discover and utilize thunderbolt0 for Ray and GPU synchronization.

Community Insights and Trade-offs

Community discussion highlights several practical considerations for this setup:

• Cost vs. Performance: While the Strix Halo provides massive unified memory (up to 256GB across two nodes), some users note that the memory bandwidth (under 300GB/s) is significantly lower than that of Apple's M-series Ultra chips (e.g., M3 Ultra at ~900GB/s), which may result in slower token generation speeds.
• Hardware Accessibility: The need for PCIe risers and external NICs makes this a complex build for those using mini-PCs rather than the Framework mainboards.
• Potential for Local AI: Despite the bandwidth gap, the ability to run 300B+ parameter models on enthusiast-grade hardware is seen as a significant milestone for local LLM deployment.