Ai2 Impactful Scheduling: GPU Time Budgets and Hierarchical Fair-Share

TL;DR

Ai2 replaced its priority‑based GPU scheduler with a budget‑driven hierarchical fair‑share system that guarantees impact‑aligned GPU time, cuts debug‑job latency from hours to seconds, and keeps cluster occupancy at 98% while reducing on‑call toil by 74%.

The scheduling problem at scale

  • Ai2 operates 88‑to‑1024‑GPU clusters of NVIDIA H100, B200, and B300 cards serving ~150 researchers across LLM, VLM, RL, and scientific workloads.
  • Demand consistently exceeds supply by a factor of 2–3, meaning each GPU hour competes with 2–3 distinct jobs.
  • The legacy priority‑based scheduler caused:
    • GPU squatting – users parked no‑op jobs to reserve capacity.
    • Priority inflation – all jobs eventually ran at HIGH priority, starving lower‑priority work.
    • High on‑call burden – engineers spent most of their time negotiating shutdowns of non‑preemptible jobs on unhealthy hosts.

From priority to budgets: the core idea

  • Instead of assigning static GPU monopolies, Ai2 allocates GPU‑time budgets to projects and researchers.
  • Managers at three hierarchy levels (lead researcher → principal investigator → program manager/CEO) set proportional shares of total capacity (e.g., Project A1 receives a 35 % claim).
  • Every workload must be funded by a budget; unfunded jobs are treated as unallocated and are fully preemptible.
  • This turns the scheduling debate into a transparent, budget‑review process rather than an ad‑hoc negotiation.

Hierarchical fair‑share scheduler

  • The scheduler implements a classic hierarchical fair‑share algorithm (originating from Hadoop Fair Scheduler, SLURM Fair Tree, YARN Fair Scheduler) but with dynamic weights derived from the budget tree.
  • It tracks GPU occupancy over a sliding 7‑day window and prioritises under‑utilised allocations over over‑utilised ones.
  • Two occupancy categories:
    1. Allocated occupancy – charged to a budget, protected from preemption for a minimum runtime.
    2. Unallocated occupancy – not charged, always preemptible, used to keep GPUs busy when budgets are idle.
  • The system automatically re‑queues resumable jobs once their protected minimum runtime expires, enabling time‑slicing across teams.

“The new scheduler makes it feel like we have an extra 30 % compute… we can later burst beyond our allocation limit and still see our jobs scheduled quickly and without preemption.” — Chris Clark

Scheduling contract: minimum runtime guarantees

  • Workloads declare a minimum runtime (the shortest period needed for meaningful progress) and whether they are resumable.
  • During this window the job is shielded from preemption; after it expires the scheduler may preempt and re‑queue it.
  • Setting the minimum runtime to zero marks the job as unallocated, making it fully preemptible and cost‑free.
  • This contract solves two pain points:
    • Prevents long‑running jobs from monopolising GPUs (reducing squatting).
    • Allows automatic draining of unhealthy hosts, cutting human‑in‑the‑loop repairs by 74 %.

Simulation framework for policy validation

  • Ai2 built a lightweight simulator that ingests historical workload submissions and runs the scheduler logic to predict queue times, preemptions, and GPU‑time distribution.
  • Simulations guided key parameters such as the 8‑hour maximum minimum runtime and verified that debug workloads would see dramatically lower latency.
  • Hand‑crafted debug‑job scenarios showed a predicted drop from ≈6 h to 5 min; real‑world rollout achieved ≈2 h to 30 s.

Production results after rollout

  • Allocation fulfillment: Teams received 98 % of the GPU hours they were owed over a 30‑day period; 13/15 teams hit ≥95 % fulfillment, worst case 90 %.
  • Cluster occupancy: Remained steady at 98 % despite 2–3× oversubscription.
  • Unallocated usage: 18 % of delivered GPU time was unallocated, providing the buffer that kept occupancy high.
  • Queue latency improvements: Median wait time on the largest H100 cluster fell from 5 min to 24 s; p90 wait dropped from 2.8 h to 1.8 h.
  • Debug jobs: p90 wait reduced from 2 h to 30 s, far exceeding simulation expectations.
  • On‑call toil: Automatic host draining reduced human‑involved repairs by 74 %.

Remaining challenges and next steps

  • Learning curve: Early confusion arose from mixed terminology and varying behavior across clusters; live explanatory sessions and updated visualizations mitigated this.
  • Interactive sessions: Time‑slicing capped protected runtimes at 8 h, causing preemptions for long dev sessions. Ai2 plans:
    1. Deploy a CPU‑only cluster for data‑prep and interactive work.
    2. Build restorable sessions that can be checkpointed and resumed on the CPU cluster.
  • Capacity fragmentation: Protecting minimum runtimes may limit the scheduler’s ability to preempt many small jobs simultaneously, potentially increasing wait times for very large workloads. Ongoing simulation work aims to quantify and address this effect.

Looking ahead: from impact to utilization

  • The next pyramid layer is utilization – ensuring that boot‑strapping, checkpointing, and training code run as efficiently as possible to maximise the value of allocated GPU time.
  • Ai2 invites collaborators to join its engineering team to tackle these utilization challenges.

This post is a faithful synthesis of the official Hugging Face blog entry “Impactful scheduling for GPU clusters” published by the Allen Institute for AI on 2026‑10‑09.

Sources