Training a foundation model, or fine-tuning a large one for days, is a capacity problem before it is a machine learning problem. The GPUs you need (p5, p5e, p5en, trn2) are the most contended instances in AWS, and a training run that loses a node at hour 40 costs you hour 40. SageMaker HyperPod exists for this, and its flexible training plans, launched in December 2024 and extended to more regions in 2025, change the cost conversation. This post is about when they pay off and how to use them well.
The capacity problem, stated plainly
For a large training job you need N accelerator instances, in one cluster placement group, for D days, starting on a date you can plan around. On-demand gives you no guarantee you will get N at once; spot gives you a discount and a guarantee you will lose some of them. Capacity reservations guarantee the instances but start the meter the moment they begin, whether your data pipeline is ready or not.
Teams respond by over-reserving (expensive), by hoarding instances they are not using (expensive and unpopular), or by accepting that a run may take three attempts (expensive and slow).
What a flexible training plan is
A flexible training plan is a reservation of a specified accelerator capacity for a specified duration within a window you choose. You tell SageMaker the instance type and count, the total compute time you need, and the earliest start and latest end; it searches for capacity and offers one or more plans (possibly split into segments) that meet the request. You pay for the reserved time at the plan's price, and SageMaker schedules your HyperPod cluster or training job onto it.
Three things make this different from a plain reservation:
- The search is done for you. You do not have to find a date when 32 p5 instances are free in one placement group; the service does, and it will propose segmented plans (for example two blocks a few days apart) when a contiguous block is not available.
- It works for both HyperPod clusters and regular SageMaker training jobs. You can attach a plan to a training job without running a HyperPod cluster at all.
- The plan is the capacity, not the job. If a run finishes early, the remaining time is available for the next job on the plan.
Training plans vs. on-demand vs. spot
| On-demand | Spot | Flexible training plan | |
|---|---|---|---|
| Capacity guarantee | None; launch may fail | None; may be reclaimed | Reserved for the plan window |
| Price | List | Deep discount | Below on-demand, above spot |
| Best for | Short jobs, small instance counts | Fault-tolerant jobs with frequent checkpoints | Multi-node, multi-day runs with a deadline |
| Failure mode | Cannot get enough instances | Interrupted mid-run | Plan search may not find your ideal window |
The decision rule we use: if the job is small enough that on-demand reliably launches, use on-demand (or spot with checkpointing for the discount). If the job needs a cluster of contended accelerators for more than a day, price a training plan first; the premium over spot is usually less than the cost of one failed attempt.
Capacity planning for a run
Before requesting a plan, you need three numbers, and getting them wrong is the most common way to waste a reservation.
- Throughput per instance. Run the training loop for 30 minutes on one or two instances of the target type and measure tokens (or samples) per second, with the real model, sequence length, and precision. Extrapolate with a scaling efficiency of 85 to 90 percent for well-configured distributed training, not 100.
- Total work. Tokens or samples per epoch times epochs, or the compute budget from your scaling-law estimate.
- Slack. Add 15 to 25 percent for evaluation steps, checkpoint writes, the occasional restart, and the first hour of any run, which never goes to plan.
Duration equals total work divided by cluster throughput, plus slack. If the number surprises you, that is the moment to reconsider the model size, sequence length, or precision, not the day the plan starts.
Checkpointing and resiliency
A plan guarantees capacity; it does not guarantee that every node stays healthy. HyperPod's value on top of the plan is resiliency: it monitors node health, replaces faulty instances, and resumes training from the last checkpoint automatically. To benefit from that, the job has to be restartable:
- Checkpoint to S3 (or Amazon FSx for Lustre mounted on the cluster) at an interval chosen from the cost of a restart, typically every 30 to 60 minutes for large runs. Asynchronous checkpoint writes keep the GPUs busy while the state is saved.
- Make the training script resume idempotently: detect the latest checkpoint at start-up, restore optimizer and scheduler state and the data loader position, and continue. Test the resume path on purpose before the real run by killing a node.
- Log per-step throughput. A node that is slow rather than dead shows up as a throughput drop long before it fails.
With those in place, a node failure costs you the interval since the last checkpoint and a few minutes of replacement time, instead of the run.
When HyperPod beats plain training jobs
A SageMaker training job with a training plan is the right tool for a single, well-defined run: one model, one script, start to finish. HyperPod is the right tool when:
- You run many jobs over weeks on the same capacity: pre-training, several fine-tunes, ablations, evaluation. HyperPod keeps the cluster and lets you schedule onto it with Slurm or Amazon EKS.
- You need automatic node recovery and fast checkpoint resume without building it yourself.
- Your researchers want persistent, SSH-able nodes with shared FSx storage, the way they work on an on-premises cluster.
- You want a task-governance layer that allocates the reserved capacity between teams by priority rather than first-come-first-served.
If none of those apply, the operational overhead of a cluster is not worth it; a training job on a plan gets you the capacity guarantee with less to manage.
Cost controls that actually work
- Size the plan from the measured throughput, not the paper estimate, and buy the slack rather than a second plan.
- Queue follow-on jobs onto the plan. Reserved time with nothing running is the purest waste in the building.
- Use mixed precision and sequence packing before buying more instances; a 20 percent throughput gain is 20 percent off the plan.
- Evaluate on cheaper instances. Evaluation and inference testing do not need the training cluster; run them on g6 instances alongside.
- Watch the egress and storage line. Checkpoints for a large model are terabytes; set lifecycle rules on the bucket.
Planning a large training run and not sure whether a plan, spot, or a standing HyperPod cluster is the right shape? Our generative AI consulting engagements start with exactly that capacity-and-cost model. Contact us.