Ai2's AI Infrastructure team runs thousands of NVIDIA H100, B200 and B300 GPUs in clusters ranging from 88 to 1024 GPUs, serving about 150 internal researchers. Demand runs 2-3x above supply at any moment, so every available GPU hour has two or three workloads competing for it. They replaced their priority-based scheduler with GPU-time budgets, hierarchical fair-share allocation and a time-slicing contract. The numbers after rollout are the interesting part.
What The Old Priority Scheduler Broke
Priority was free, and workloads could opt out of preemptability. That produced three predictable pathologies. GPU squatting: users parked no-op workloads they could connect to when the need arose, because they could not launch a debug job with low enough latency. Priority inflation: eventually 100% of scheduled workloads declared HIGH priority, starving every lower level. And on-call toil: because preemptability was optional, engineers spent most of their ticket time negotiating shutdowns of non-preemptable workloads on hosts with known maintenance problems.
Ai2's own framing is a tragedy of the commons. They were manually solving a knapsack problem, fitting dynamically changing research needs into a static schedule. Assigning teams GPU monopolies gave the ownership incentive but left GPUs idle because research demand is seasonal — one team's quiet week is another team's crunch.
Budgets Instead Of GPUs, And A Minimum-Runtime Contract
The replacement allocates GPU time rather than hardware. Managers proportionally assign shares down a program tree — one example in the post gives project A1 a 35% claim on total capacity. Every request for protected GPU time must be funded by a budget, or it is not protected from preemption. A squatting workload now spends its team's budget on nothing, which makes gaming the scheduler more expensive than arguing for a real budget increase.
The fair-share algorithm itself is not new — it descends from the Hadoop Fair Scheduler (2009) and is in active use in SLURM's Fair Tree and YARN's Fair Scheduler. What is new is the input: the tree mirrors the research program structure and the weights are manager-set budgets instead of static quotas. The scheduler tracks occupancy over a sliding lookback window, defaulting to 7 days, and sorts workloads from under-utilized allocations above over-utilized ones.
Two kinds of occupancy matter:
| Type | Charged to budget | Preemptible |
|---|---|---|
| Allocated | Yes | No, during minimum runtime |
| Unallocated | No | Always |
Unallocated occupancy is what keeps GPUs busy when funded work is not ready to run. Teams can never decline free GPU cycles because those cycles cost them nothing.
The scheduling contract adds time-slicing. A workload declares its minimum runtime — the shortest occupancy needed to make meaningful progress — and is protected from preemption for that window. Minimum runtime can be set to zero, which marks the GPU time unallocated. In exchange, the scheduler may rebalance once the progress is banked, automatically re-queueing resumable workloads. Ai2 capped the maximum protected runtime at 8 hours. Crucially, unhealthy hosts can drain workloads as they reach minimum runtime, which cut repairs requiring a human in the loop by 74%.
The Measured Results
Since the cluster-by-cluster rollout began at the end of July:
- Teams received 98% of the GPU hours they were owed over a 30-day test period; 13 of 15 team allocations got 95% or more, worst case 90%.
- Cluster occupancy held steady at 98% before and after, with demand still 2-3x capacity in both periods.
- 18% of delivered GPU time was unallocated, which is how occupancy stayed high when funded use cases were not ready.
- Debug workload p90 queue wait fell from 2 hours to 30 seconds in production, against a simulator prediction of 6 hours to 5 minutes.
- On the largest H100 cluster, median queue wait fell from 5 minutes to 24 seconds, and p90 wait dropped about a third, from 2.8 hours to 1.8 hours.
They built a small simulator first, taking workloads and submission schedules as input and jumping ahead to schedulable moments to report queue waits, preemption events and GPU-time distribution across projects — many simulated days in a few seconds. They used it to test the lookback window length and the maximum minimum-runtime value before committing to 8 hours.
What Did Not Get Better
Interactive sessions regressed. Researchers used to hold an analysis or code-writing session for up to a week; under time-slicing they hit the 8-hour protected cap, and being preempted meant waiting for a new session and rebuilding state by hand. Ai2 is now building a CPU-only cluster next to on-prem storage for data-prep dev sessions, plus restorable sessions so a preempted session can resume elsewhere.
They are also watching capacity fragmentation. Minimum-runtime protection now covers jobs that used to rely on preemptibility to exceed team limits, so the scheduler may have fewer chances to interrupt many jobs at once and place a large pending workload. That is currently being reproduced in the simulator against production ground truth.
What I Would Take From This
The transferable idea is not the fair-share algorithm — that is two decades old and already in SLURM and YARN. It is the contract. Requiring a workload to declare a funded minimum runtime converts "who deserves the GPUs" from an on-call negotiation into an administrative budgeting decision, and it makes idle capacity free to anyone rather than exclusive to whoever squatted first. If you run a shared scheduler of any kind, the two knobs worth stealing are the sliding lookback window and a hard cap on protected runtime. The catch is exactly what Ai2 hit: long-lived interactive sessions with volatile state need somewhere else to live, and that is a real project, not a config change.
Sources: