GPU Infrastructure & Scheduling

GPU Infrastructure & Scheduling#

A team at a mid-sized AI startup ran a distributed training job for 47 hours. Somewhere around $50,000 in cloud GPU costs, a higher-priority job arrived on the cluster. The scheduler preempted their run. No checkpointing was configured. They started from scratch.

That story isn't unusual. As models have grown from millions to hundreds of billions of parameters, GPU infrastructure has gone from "something the ...