vime ROCm Support for AMD Instinct GPUs

vime ROCm Support for AMD Instinct GPUs

vLLM has integrated ROCm support into vime, its reinforcement learning (RL) framework, allowing large-scale RL post-training to run natively on AMD Instinct MI300X and MI355X GPUs. This integration provides a validated end-to-end pipeline, including a prebuilt container to eliminate the need for building from source.

vime Architecture and Design

vime utilizes a three-stage, decoupled train-inference design based on the slime framework. It separates the RL process into three distinct components:

  • Training (Megatron): Manages the main training loop, parameter updates, and weight synchronization to the rollout side.
  • Rollout (vLLM + Router): Handles inference sampling to produce training samples based on reward or verifier signals.
  • Data Buffer: Acts as the bridge between training and rollout, managing custom rollout logic and prompt injection.

Hardware Advantages of AMD Instinct GPUs for RL

RL post-training is highly memory-intensive because it requires simultaneous storage of training-side weights (Megatron format) and inference-side KV caches (vLLM rollouts). AMD Instinct GPUs are optimized for this profile:

  • High HBM Capacity: The MI300X offers 192 GB of HBM3, while the MI355X provides 288 GB. This large capacity reduces the need for aggressive tensor parallelism, simplifying topology and increasing cluster utilization.
  • Superior Memory Bandwidth: The MI300X delivers over 5 TB/s and the MI355X delivers 8 TB/s of aggregate bandwidth. Since RL rollouts are memory-bandwidth-bound, this higher throughput reduces step latency during the autoregressive token generation phase.
  • Native Ecosystem Integration: ROCm's open-source nature allows vime to inherit the existing vLLM rollout stack without requiring a separate code path, leveraging native PyTorch and vLLM support for ROCm.

Technical Implementation on ROCm

Bringing vime to AMD hardware involved the integration of several key software components:

Megatron-LM Backend

vime uses a ROCm-compatible fork of Megatron-LM. To ensure compatibility, a patch was implemented to guard CUDA fused-kernel initialization on non-CUDA builds. The pipeline supports HuggingFace to torch_dist checkpoint conversion on a single GPU and utilizes the native PyTorch path for gradient accumulation.

Colocated Weight Synchronization

In colocated mode, Megatron and vLLM share the same GPU pool. vime uses Inter-Process Communication (IPC) to synchronize updated weights from Megatron to vLLM after each optimizer step. This is enabled on ROCm by the torch.cuda.get_device_properties(i).uuid interface, which provides stable, process-consistent device UUIDs for routing.

Resource Management and Ray Integration

GPU assignment is managed via HIP_VISIBLE_DEVICES. vime ensures consistency between the Megatron training actor and vLLM subprocesses by setting this alongside CUDA_VISIBLE_DEVICES. Ray's AMD GPU manager is configured to respect these masks, and the system requires a raised file descriptor limit (--ulimit nofile=1048576:1048576) to spawn actor workers at scale.

Performance Benchmarks on MI355X

Testing with the Qwen3-8B model on MI355X GPUs demonstrated the following results over 100 training steps:

  • Throughput: Sustained approximately 4,100 tokens_per_gpu_per_second. Throughput trended upward as the policy learned more predictable outputs, allowing vLLM to batch more efficiently.
  • Logprob Stability: The train_rollout_logprob_abs_diff remained steady around 0.012. This low divergence between training and rollout log probabilities indicates successful weight synchronization and is comparable to results on NVIDIA hardware.
  • Reward Convergence: Using the dapo-math-17k dataset, the raw_reward climbed from near 0 to approximately 0.5–0.6, signaling that the policy successfully learned to favor reasoning patterns rewarded by the verifier.

Supported Features and Roadmap

Current Support

  • GRPO training
  • Colocated and asynchronous (disjoint GPU pool) training and rollout
  • Megatron-LM training backend and vLLM rollout backend
  • Support for Qwen3 Dense and MoE models

Future Roadmap

  • Full vLLM Router and PD (Prefill-Decode) disaggregation support
  • FP8 pipeline optimization
  • R3 (Rollout Routing Replay) for AMD MoE workloads
  • Performance optimizations for asynchronous training to address memory leaks and logprob divergence
  • Agentic RL for multi-agent settings and multi-turn tool calling

Sources