Who Gets the GPUs? HyperPod Task Governance on EKS for Shared Clusters

Most teams that buy reserved GPU capacity end up with the same complaint within a quarter: the cluster shows 95 percent allocated and 30 percent utilized. One team holds eight H100s for a notebook that has been idle since Tuesday, another team's fine-tuning job has been pending for two days, and nobody can point at a rule that decides who wins.

SageMaker HyperPod task governance is the piece that fixes this on EKS-orchestrated HyperPod clusters. It gives each team a compute allocation, lets idle capacity be borrowed by other teams, and preempts borrowed capacity when the owner comes back. This tutorial sets it up end to end: cluster prerequisites, team quotas, submitting a governed training task, and the dashboards you use to prove the change worked.

What task governance actually does

Three mechanics, and it is worth being precise about them because they drive every configuration decision:

• Allocation. Each team gets a quota expressed in accelerators (or instances) per instance type. A team can always run up to its quota.
• Lending and borrowing. A team can mark unused quota as lendable. Other teams over their own quota can borrow it, so the cluster runs hot instead of reserved-and-idle.
• Preemption. When the lending team submits work and needs its quota back, borrowed tasks are preempted by priority class, lowest first. Preempted tasks return to the queue; they do not fail.

On top of that sits task prioritization: named priority classes with weights, so an inference-serving task can jump ahead of an exploratory tuning sweep inside the same team's quota.

Two things it is not: it is not billing chargeback (it governs scheduling, not invoices), and it is not available on Slurm-orchestrated HyperPod clusters. If your cluster is Slurm, your equivalent tooling is Slurm partitions and QOS, and the rest of this guide will not apply.

Prerequisites

1. A HyperPod cluster with EKS orchestration. Check with:

   aws sagemaker describe-cluster --cluster-name ml-shared-01 \
     --query 'Orchestrator'
   

   An Eks.ClusterArn in the response means you are on the right path.

2. The HyperPod task governance EKS add-on installed on the cluster:

   aws eks create-addon \
     --cluster-name ml-shared-01-eks \
     --addon-name amazon-sagemaker-hyperpod-taskgovernance
   

   Verify it reaches ACTIVE:

   aws eks describe-addon --cluster-name ml-shared-01-eks \
     --addon-name amazon-sagemaker-hyperpod-taskgovernance \
     --query 'addon.status'
   

3. kubectl access to the EKS cluster plus the HyperPod CLI:

   aws eks update-kubeconfig --name ml-shared-01-eks
   pip install sagemaker-hyperpod
   hyp list-cluster
   

4. Instance groups sized for real work. Governance can only schedule what exists; it does not create capacity. If you also buy capacity through flexible training plans, read Controlling Foundation-Model Training Costs with HyperPod Flexible Training Plans first, because the quota math below should sum to the capacity your plan actually delivers.

Step 1: Design the allocation before you type anything

The mistake is to divide the cluster equally. Allocate against observed demand and leave a shared pool. A workable starting split for a 32-GPU cluster shared by three teams:

Team	Quota	Lendable	Rationale
platform-mlops	4 GPUs	Yes, 100%	Bursty CI and evaluation jobs, long idle stretches
research	14 GPUs	Yes, 50%	Continuous experimentation, tolerant of preemption
product-genai	10 GPUs	No	Deadline-bound fine-tunes, must not be preempted
unallocated buffer	4 GPUs	—	Absorbs onboarding and one-off spikes

Rules of thumb that hold up in practice:

• Never allocate 100 percent of the cluster to named teams. A 10 to 15 percent buffer stops every new project from becoming a quota renegotiation.
• Teams that own production deadlines should lend little or nothing. Teams doing exploration should lend aggressively; preemption is cheap for them if they checkpoint.
• Set quota in accelerators, not instances, unless a team needs whole-node placement for multi-node NCCL jobs. Then allocate in instances so they get full nodes.

Step 2: Create compute allocations

Console path: SageMaker AI console, HyperPod clusters, your cluster, Policies, then Compute allocation. The API equivalent is a cluster scheduler config, which is the version you want in Git:

{
  "ClusterArn": "arn:aws:sagemaker:us-east-1:111122223333:cluster/ml-shared-01",
  "Name": "shared-cluster-policy",
  "SchedulerConfig": {
    "PriorityClasses": [
      { "Name": "inference-serving", "Weight": 100 },
      { "Name": "production-training", "Weight": 75 },
      { "Name": "experimentation", "Weight": 25 }
    ],
    "FairShare": "Enabled"
  }
}

aws sagemaker create-cluster-scheduler-config \
  --cli-input-json file://scheduler-config.json

Then one quota per team:

aws sagemaker create-compute-quota \
  --name research-quota \
  --cluster-arn arn:aws:sagemaker:us-east-1:111122223333:cluster/ml-shared-01 \
  --compute-quota-target '{"TeamName":"research","FairShareWeight":50}' \
  --compute-quota-config '{
      "ComputeQuotaResources":[{"InstanceType":"ml.p5.48xlarge","Count":2,"Accelerators":14}],
      "ResourceSharingConfig":{"Strategy":"LendAndBorrow","BorrowLimit":50},
      "PreemptTeamTasks":"LowerPriority"
    }' \
  --activation-state Enabled

The three fields that matter most:

• Strategy: LendAndBorrow, Lend (share out but never borrow), or DontLend for the team that must not be touched.
• BorrowLimit: percentage above the team's own quota it may borrow. Leaving this unbounded lets one team's sweep swallow the cluster the moment it goes quiet elsewhere.
• PreemptTeamTasks: whether a team's own lower-priority tasks can be preempted to make room for its higher-priority ones. Set it to LowerPriority for teams that run both serving and batch work.

Each quota maps to a Kubernetes namespace of the form hyperpod-ns-<team>, with a queue created for it. Grant team IAM principals access to their namespace only; governance without RBAC is a suggestion, not a policy.

Step 3: Submit a governed task

Tasks must carry a queue and a priority label or the scheduler cannot place them. With the HyperPod CLI:

hyp create hyp-pytorch-job \
  --job-name llama-8b-sft-run14 \
  --image 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.6.0-gpu-py312 \
  --command '["torchrun","--nproc_per_node=8","train.py"]' \
  --instance-type ml.p5.48xlarge \
  --node-count 2 \
  --namespace hyperpod-ns-research \
  --queue-name hyperpod-ns-research-localqueue \
  --priority experimentation

The equivalent labels on a raw PyTorchJob, if you submit through your own manifests or Kueue-aware tooling:

metadata:
  namespace: hyperpod-ns-research
  labels:
    kueue.x-k8s.io/queue-name: hyperpod-ns-research-localqueue
    kueue.x-k8s.io/priority-class: experimentation

Watch placement and preemption decisions:

hyp list-jobs --namespace hyperpod-ns-research
kubectl get workloads -n hyperpod-ns-research
kubectl describe workload <name> -n hyperpod-ns-research

The describe workload events are where you find out why something is pending: quota exhausted, borrow limit hit, or waiting on a preemption to drain.

Step 4: Make preemption survivable

Governance will kill borrowed tasks. That is the point, and it is only acceptable if your training code checkpoints and resumes. Minimum bar:

• Write checkpoints to /opt/ml/checkpoints (backed by S3 or FSx) on a fixed step interval, not only at epoch end.
• On startup, look for the newest checkpoint and resume optimizer and scheduler state, not just weights.
• Keep checkpoint writes asynchronous so the interval can be tight without wrecking throughput.
• Handle SIGTERM by flushing a checkpoint before the grace period expires.

import os, signal, torch

CKPT_DIR = os.environ.get("SM_CHECKPOINT_DIR", "/opt/ml/checkpoints")

def save(step, model, optimizer, scheduler):
    tmp = f"{CKPT_DIR}/tmp-{step}.pt"
    torch.save({
        "step": step,
        "model": model.state_dict(),
        "optimizer": optimizer.state_dict(),
        "scheduler": scheduler.state_dict(),
    }, tmp)
    os.replace(tmp, f"{CKPT_DIR}/latest.pt")  # atomic: never a half-written latest

def install_preemption_hook(state):
    def handler(signum, frame):
        save(state["step"], state["model"], state["optimizer"], state["scheduler"])
        raise SystemExit(0)
    signal.signal(signal.SIGTERM, handler)

Teams whose jobs cannot resume should be on DontLend quota and a high priority class, and should expect a smaller allocation in exchange. That trade is the conversation governance is designed to make explicit.

Step 5: Watch utilization, not allocation

The observability dashboard in the SageMaker AI console shows per-team allocation versus actual accelerator utilization, task queue depth, and preemption counts. The same metrics land in Amazon Managed Service for Prometheus if you enabled the metrics add-on, so you can alert on them.

Four numbers worth reviewing weekly:

1. Cluster accelerator utilization. If allocation is high and utilization is low, quotas are too generous or idle interactive workloads are holding GPUs.
2. Queue wait time, p90, per team. Rising wait for a team that never borrows means its quota is genuinely too small.
3. Preemption rate. A high rate is healthy borrowing; a high rate against non-checkpointing jobs is wasted compute.
4. Idle-but-allocated GPU hours. The clearest input to the next quota review, and the number that justifies governance to whoever signs for the capacity.

A quarterly quota review using those four numbers takes an hour and removes almost all of the political argument, because the allocation moves toward whoever demonstrably uses it.

Common failure modes

• Tasks stay pending forever. Almost always a missing or wrong queue-name label, or a quota that names an instance type the cluster does not have. Check kubectl describe workload.
• One team borrows everything. No BorrowLimit set. Cap it at 50 to 100 percent of own quota.
• Production job preempted. The team was on LendAndBorrow with a low priority class. Production work belongs in a high-weight priority class on DontLend quota.
• Governance add-on installed, nothing enforced. Teams are still submitting into default or kube-system namespaces. Lock namespace access down with RBAC and IAM.
• Quotas sum to more than the cluster. Legal, but then every team is effectively borrowing and preemption becomes constant noise. Keep the sum at or below physical capacity plus your intended buffer.

Where this fits

Task governance is a cluster-level control. It pairs with the job-level cost work we cover in Multi-Node FSDP Training on SageMaker AI and, on the serving side, with Inference Components and Scale-to-Zero. Getting all three right is usually the difference between a GPU cluster that looks expensive and one that looks justified.

Running a shared GPU cluster where the quota argument keeps coming back? Talk to us — we set up HyperPod governance policies, checkpoint-safe training code, and the utilization reporting that keeps the allocation honest.