Most SageMaker AI bills do not blow up because of one expensive training run. They blow up because nobody can answer "which team owns this?" fast enough. A Studio space left running on an ml.g5.4xlarge, a proof-of-concept endpoint from a demo two quarters ago, a processing job that someone scheduled hourly and forgot — individually small, collectively the majority of the waste we find when we are brought in to look at an ML account.
This tutorial sets up the account-level plumbing that makes SageMaker AI spend attributable and self-limiting: a tag standard that actually reaches the billing data, cost allocation you can slice by team and project, automatic shutdown of idle Studio resources, a nightly sweep for orphaned endpoints, and budget alarms that fire before the invoice does.
This is the layer underneath per-workload optimisation. If you have not yet squeezed the workloads themselves, read Inference Components and Scale-to-Zero and Faster Tokens, Same GPU as well. Governance tells you where the money goes; optimisation reduces it.
Step 1: agree a tag standard and make it non-optional
Pick a small set of keys. Long tag taxonomies die. Four is usually enough:
| Key | Example | Why |
|---|---|---|
Owner | jsmith@example.com | Someone to email before deleting |
Project | churn-model | The unit of chargeback |
Environment | dev / staging / prod | Lets you apply harsher rules to dev |
ExpiresOn | 2026-09-30 | The single most useful tag on a dev endpoint |
SageMaker propagates tags from a training job, model, endpoint config, or endpoint onto the underlying billed usage, so tags applied at creation time reach Cost Explorer. Tags added after creation do not retroactively re-tag past usage — which is why the enforcement below matters more than a clean-up sweep.
With the Python SDK, tags go on at creation:
import sagemaker
from sagemaker.huggingface import HuggingFaceModel
TAGS = [
{"Key": "Owner", "Value": "jsmith@example.com"},
{"Key": "Project", "Value": "churn-model"},
{"Key": "Environment", "Value": "dev"},
{"Key": "ExpiresOn", "Value": "2026-09-30"},
]
model = HuggingFaceModel(env=hub, role=role, transformers_version="4.49",
pytorch_version="2.6", py_version="py312")
predictor = model.deploy(
initial_instance_count=1,
instance_type="ml.g5.2xlarge",
endpoint_name="churn-scorer-dev",
tags=TAGS,
)
Make it non-optional with an IAM condition on the execution role that data scientists assume. aws:RequestTag checks the tags supplied on the create call:
{
"Sid": "DenyUntaggedSageMakerResources",
"Effect": "Deny",
"Action": [
"sagemaker:CreateEndpoint",
"sagemaker:CreateTrainingJob",
"sagemaker:CreateProcessingJob",
"sagemaker:CreateTransformJob",
"sagemaker:CreateNotebookInstance"
],
"Resource": "*",
"Condition": {
"Null": {
"aws:RequestTag/Project": "true",
"aws:RequestTag/Owner": "true"
}
}
}
Roll this out in stages. Attach it to one team first, in a dev account, and give people a week of noisy AccessDenied before you widen it — otherwise you will spend that week unblocking pipelines instead of saving money.
A gentler complement is an AWS Config rule (required-tags) scoped to SageMaker resource types, reporting non-compliance instead of blocking. Use Config for visibility and IAM for the resource types that are genuinely expensive: endpoints, training jobs, and notebook instances.
Step 2: activate the tags as cost allocation keys
This is the step teams skip, and it silently wastes the whole tagging effort. Tags do not appear in Cost Explorer or the Cost and Usage Report until you activate them as cost allocation tags in the payer account:
Billing and Cost Management → Cost allocation tags → User-defined cost allocation tags → select
Owner,Project,Environment,ExpiresOn→ Activate.
Two gotchas: activation only affects usage from roughly the start of the current month onward, so do it now rather than the day before a review; and it must be done in the management account of the organisation, not the member account where the workloads run.
Once active, you can query spend per project from the CLI:
aws ce get-cost-and-usage \
--time-period Start=2026-02-01,End=2026-03-01 \
--granularity MONTHLY \
--metrics UnblendedCost \
--filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon SageMaker"]}}' \
--group-by Type=TAG,Key=Project
Swap Key=Project for Type=DIMENSION,Key=USAGE_TYPE to see the split between training (ML-Train), real-time hosting (ML-Host), Studio notebooks (ML-Notebook), and processing. In most accounts we audit, hosting is 60–80% of SageMaker spend and almost all of the surprise, because endpoints bill by the instance-hour regardless of traffic.
Step 3: shut down idle Studio resources automatically
Studio spaces are the classic leak: someone opens a GPU-backed JupyterLab space on Friday afternoon and it runs all weekend. SageMaker AI supports idle shutdown as a lifecycle setting on the domain or user profile, so you do not need a custom lifecycle config script any more.
Set it at the domain level so it applies to everyone by default:
aws sagemaker update-domain \
--domain-id d-xxxxxxxxxxxx \
--default-user-settings '{
"JupyterLabAppSettings": {
"AppLifecycleManagement": {
"IdleSettings": {
"LifecycleManagement": "ENABLED",
"IdleTimeoutInMinutes": 60,
"MinIdleTimeoutInMinutes": 60,
"MaxIdleTimeoutInMinutes": 480
}
}
}
}'
IdleTimeoutInMinutes is the default; the min/max bound what a user can change it to in the Studio UI. Sixty minutes for JupyterLab and code editor spaces is a reasonable starting point — long enough that a lunch break does not kill a kernel, short enough that a forgotten space costs a few dollars rather than a few hundred. Apply the same block under CodeEditorAppSettings for VS Code spaces.
Important: idle shutdown stops the app. The space's EBS volume persists and continues to bill at storage rates, which is what you want (work is not lost) but is worth noting when you compare before/after numbers. Deleting unused spaces entirely is a separate, quarterly job.
If you are still on classic notebook instances, they have no native idle timeout. Migrate them, or attach a lifecycle configuration that polls the Jupyter API and calls stop-notebook-instance.
Step 4: sweep for orphaned and expired endpoints
Endpoints have no idle timeout at all. A nightly Lambda that reads the ExpiresOn tag and cross-checks CloudWatch Invocations catches both expired dev endpoints and prod endpoints nobody actually calls.
import boto3, datetime
sm = boto3.client("sagemaker")
cw = boto3.client("cloudwatch")
sns = boto3.client("sns")
TOPIC = "arn:aws:sns:us-east-1:111122223333:ml-cost-alerts"
today = datetime.date.today()
def invocations_last_7d(endpoint_name):
end = datetime.datetime.utcnow()
resp = cw.get_metric_statistics(
Namespace="AWS/SageMaker",
MetricName="Invocations",
Dimensions=[
{"Name": "EndpointName", "Value": endpoint_name},
{"Name": "VariantName", "Value": "AllTraffic"},
],
StartTime=end - datetime.timedelta(days=7),
EndTime=end,
Period=86400,
Statistics=["Sum"],
)
return sum(p["Sum"] for p in resp["Datapoints"])
def lambda_handler(event, context):
findings = []
paginator = sm.get_paginator("list_endpoints")
for page in paginator.paginate(StatusEquals="InService"):
for ep in page["Endpoints"]:
name, arn = ep["EndpointName"], ep["EndpointArn"]
tags = {t["Key"]: t["Value"] for t in
sm.list_tags(ResourceArn=arn)["Tags"]}
expires = tags.get("ExpiresOn")
if expires and datetime.date.fromisoformat(expires) < today:
findings.append(f"EXPIRED {name} (ExpiresOn={expires}, "
f"owner={tags.get('Owner','unknown')})")
continue
if invocations_last_7d(name) == 0:
findings.append(f"IDLE 7d {name} "
f"(owner={tags.get('Owner','unknown')})")
if findings:
sns.publish(
TopicArn=TOPIC,
Subject="SageMaker endpoint sweep",
Message="\n".join(findings),
)
return {"findings": len(findings)}
Run it on an EventBridge schedule (cron(0 7 * * ? *)) with an execution role allowing sagemaker:ListEndpoints, sagemaker:ListTags, cloudwatch:GetMetricStatistics, and sns:Publish.
Report first, delete later. Once the team trusts the report — give it a month — you can extend it to call delete_endpoint automatically for anything tagged Environment=dev that is past ExpiresOn, while leaving prod as a notification only. Never auto-delete production inference from a cron job; the one time the tag is wrong will cost far more than the instance-hours saved.
The same pattern applies to list_transform_jobs, list_processing_jobs with StatusEquals="InProgress" for jobs that hang, and to list_apps for Studio spaces that predate your idle settings.
Step 5: budgets that alert before the invoice
Cost Explorer is retrospective. AWS Budgets is the thing that emails you on the 9th of the month. Create one budget per project tag, plus an account-level SageMaker budget:
aws budgets create-budget \
--account-id 111122223333 \
--budget '{
"BudgetName": "sagemaker-churn-model",
"BudgetLimit": {"Amount": "4000", "Unit": "USD"},
"TimeUnit": "MONTHLY",
"BudgetType": "COST",
"CostFilters": {
"Service": ["Amazon SageMaker"],
"TagKeyValue": ["user:Project$churn-model"]
}
}' \
--notifications-with-subscribers '[{
"Notification": {
"NotificationType": "FORECASTED",
"ComparisonOperator": "GREATER_THAN",
"Threshold": 100,
"ThresholdType": "PERCENTAGE"
},
"Subscribers": [{"SubscriptionType":"SNS",
"Address":"arn:aws:sns:us-east-1:111122223333:ml-cost-alerts"}]
}]'
Use FORECASTED rather than ACTUAL for the first threshold — a forecast breach on day 6 is actionable, an actual breach on day 27 is a post-mortem. Add a second ACTUAL notification at 80% as a backstop, and enable AWS Cost Anomaly Detection with a monitor scoped to the SageMaker service; it catches the shape of spend changing (a new p5 instance type appearing) even when the total is still under budget.
Step 6: commit to the baseline, not the peak
Once attribution and clean-up are in place, and only then, look at commitments. SageMaker Savings Plans cover ML instance usage across training, real-time hosting, processing, and Studio notebooks in exchange for a one- or three-year hourly commitment, at a meaningful discount to on-demand.
Two rules from experience:
- Commit to the floor of your last 90 days of usage, not the average. Under-committing costs you a smaller discount; over-committing costs you cash for a year.
- Exclude anything you plan to migrate. If you are moving inference to Inferentia2 or consolidating onto inference components this quarter, your GPU baseline is about to drop — commit after the migration, not before.
Spot capacity via managed spot training and HyperPod flexible training plans is the complementary lever on the training side; we cover both in Multi-Node FSDP Training and Controlling Foundation-Model Training Costs.
A 30-day rollout order
- Week 1 — Agree the four tag keys. Activate cost allocation tags in the management account (do this on day one; the data starts accruing from the current month).
- Week 2 — Deploy the endpoint sweep Lambda in report-only mode. Circulate the first report; expect it to be embarrassing.
- Week 3 — Enable Studio idle shutdown at the domain level. Attach the tag-enforcement IAM policy to one team.
- Week 4 — Create per-project budgets with forecast alerts and a Cost Anomaly Detection monitor. Review the first month of tagged Cost Explorer data and decide on Savings Plans.
The order matters. Budgets without tags tell you the account is over, not who to talk to. Savings Plans before clean-up commit you to a year of the waste you were about to delete.
Where this usually lands
In the accounts we have reviewed, the split of recoverable spend is fairly consistent: roughly half from idle or orphaned real-time endpoints, a quarter from Studio spaces and notebook instances left running, and the rest from oversized training instances and duplicated data in S3 and Feature Store. None of that requires a model change — it requires knowing who owns what.
Want an outside read on where your SageMaker AI spend is going, or help putting this in place without stalling the data science team? Talk to us.