Dharma AI GPU Management: Increasing Cluster Utilization via Constraint-Aware Allocation

Dharma AI has introduced a constraint-aware GPU allocator designed to optimize cluster utilization and priority-weighted output. By changing the order of allocation decisions rather than the hardware, the system increased GPU utilization by up to 33 percentage points and priority-weighted output by up to 105% across seven benchmark scenarios when compared to a First-In-First-Out (FIFO) scheduler.

The Core Problem: Incompatible Workload Shapes

GPU scheduling is complicated by the competition between two fundamentally different allocation shapes:

Batch-like Workloads: Training, batch inference, and quantization. These require contiguous blocks of GPUs held without interruption until completion.
Elastic Workloads: Real-time inference, which is driven by a demand curve that fluctuates every timestep.

When these incompatible shapes compete for the same hardware, traditional FIFO schedulers create significant inefficiencies. FIFO schedulers often rely on fixed reservations for real-time inference to guarantee availability, meaning GPUs are held during traffic troughs and remain unavailable to batch jobs, leading to low utilization.

Constraint-Aware Allocation vs. FIFO

FIFO scheduling costs capacity in two primary ways: through static reservations that leave GPUs idle during non-peak hours and through an ordering process that ignores job priority and future capacity needs.

In contrast, the Dharma AI allocator treats real-time demand as a curve rather than a ceiling. It allocates GPUs based on demand at each timestep and allows batch-like work to occupy the troughs. To ensure safety, the system enforces a cap on how many GPUs a real-time job can swap between consecutive timesteps.

Performance Benchmarks

Across five high-contention scenarios, the allocator improved both utilization and priority-weighted value:

Utilization: Increased from a 52–85% band to a 72–88% band.
Priority-Weighted Value: Rose between 24.6% and 105.1%, with an average increase of 52%.

In a training-heavy workload on 8 GPUs, utilization rose from 53.6% to 87.0%, and value increased by 105%. Notably, in a scale test with 64 GPUs and 30 jobs, the allocator delivered 15.9% more priority-weighted value despite having identical utilization (44.9%) and throughput (27/30 jobs) as the FIFO baseline, demonstrating that priority-aware placement is critical for value generation.

Technical Implementation and Constraints

To avoid the limitations of local heuristic rules, the allocator uses a formal model to define legal allocations based on five constraints:

A GPU serves at most one job per timestep.
Every job respects its demand range, and running work is inherited and held.
Batch-like jobs occupy contiguous blocks of GPUs (sized to a power of two).
Real-time jobs have a hard cap on GPU swaps between consecutive timesteps.
Started jobs cannot be interrupted.

The Objective Function

The system uses an objective function with two terms: a reward for allocating GPUs to batch-like jobs (priority multiplied by a time-decay weight) and a penalty for failing to meet real-time demand. The real-time penalty is weighted 5 to 10 times higher than the allocation reward, ensuring latency obligations are prioritized over batch work without requiring static reservations.

Heuristic Execution

Because combinatorial allocation is NP-hard, the system employs a heuristic on the hot path to ensure low latency. The allocator runs in 1 to 2 milliseconds for contended scenarios and 15 milliseconds for a 64-GPU cluster, making it fast enough to run on every incoming request.

Demand Forecasting and Specialization

The allocator's effectiveness depends on accurate demand predictions. Dharma AI utilizes specialized estimators for different workload types:

Training: A forecaster using 22 features, including 10 concrete training variants (e.g., LoRA vs. full fine-tuning), to account for massive differences in GPU memory and duration.
Quantization: A dedicated forecast based on calibration tiers by parameter count and algorithm (e.g., AWQ, GPTQ).
Real-time Inference: A continuously recalibrated weekly demand profile rebuilt from hourly traffic history.

Operational Strategy: Optimize the Day, Commit the Hour

To mitigate forecast errors, the scheduler optimizes a 24-hour horizon but only commits the current timestep. The system re-runs every 30 to 60 minutes, allowing it to absorb forecast errors through re-optimization rather than compounding them. This approach prevents the "end-of-world effect," where an optimizer makes poor present decisions because it cannot see beyond its horizon.