Most SageMaker AI inference bills are not big because the models are big. They are big because a ml.g5.2xlarge sat at 4 percent utilization all weekend, and because three models that would happily share one GPU each got their own endpoint. Inference components and scale-to-zero endpoints fix both problems, and they are the two features we reach for first on almost every cost review.
This tutorial shows how to pack several models onto one endpoint with inference components, how to let a component scale down to zero replicas when traffic stops, and what changes in your client code and your latency expectations when you do.
Everything below uses the SageMaker Python SDK (pip install "sagemaker>=2.230") plus a couple of boto3 calls where the SDK does not expose a knob.
The mental model
Before inference components, an endpoint was: endpoint -> endpoint config -> one or more production variants, each variant being a model plus an instance type plus a count. Scaling meant scaling instances, and the floor was one instance per variant, forever.
With inference components, the layering changes:
- The endpoint owns the instances (a managed instance pool with a min and max count).
- Each inference component is a model plus its own compute request (CPU, memory, accelerator count), its own replica count, and its own autoscaling policy.
- Several components share the endpoint's instances. SageMaker AI places replicas onto instances for you.
That gives you two independent dials: how many instances the endpoint holds, and how many replicas each model runs. It also gives you the thing that actually saves money — a component can go to zero replicas, and if all components on the endpoint are at zero, the endpoint drops to zero instances and you stop paying for compute while the endpoint object still exists.
Step 1: create an endpoint with a managed instance pool
The important part is ManagedInstanceScaling and RoutingConfig. Least-outstanding-requests routing matters much more than usual here, because replicas of an LLM component have very uneven per-request cost.
import boto3, sagemaker
sm = boto3.client("sagemaker")
role = sagemaker.get_execution_role()
sm.create_endpoint_config(
EndpointConfigName="shared-gpu-config",
ExecutionRoleArn=role,
ProductionVariants=[
{
"VariantName": "AllTraffic",
"InstanceType": "ml.g5.12xlarge", # 4 x A10G
"ModelDataDownloadTimeoutInSeconds": 1200,
"ContainerStartupHealthCheckTimeoutInSeconds": 1200,
"ManagedInstanceScaling": {
"Status": "ENABLED",
"MinInstanceCount": 0, # 0 enables scale to zero
"MaxInstanceCount": 3,
},
"RoutingConfig": {"RoutingStrategy": "LEAST_OUTSTANDING_REQUESTS"},
}
],
)
sm.create_endpoint(
EndpointName="shared-gpu",
EndpointConfigName="shared-gpu-config",
)
Two details people trip over:
MinInstanceCount: 0is what makes scale to zero possible at all. If you leave it at 1 you keep an instance warm no matter what your component policies say.- The generous startup timeouts are not paranoia. When an endpoint wakes from zero it must launch an instance and pull a model artifact; a 7B model on a cold instance can take several minutes.
Step 2: register models and attach them as components
Create the SageMaker model objects as usual, then attach each one as a component with an explicit compute request.
from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri
llm = HuggingFaceModel(
role=role,
image_uri=get_huggingface_llm_image_uri("huggingface"),
env={
"HF_MODEL_ID": "meta-llama/Llama-3.1-8B-Instruct",
"SM_NUM_GPUS": "1",
"MAX_INPUT_TOKENS": "4096",
"MAX_TOTAL_TOKENS": "8192",
},
)
llm.create(instance_type="ml.g5.12xlarge") # registers the Model, no endpoint
sm.create_inference_component(
InferenceComponentName="llama-31-8b",
EndpointName="shared-gpu",
VariantName="AllTraffic",
Specification={
"ModelName": llm.name,
"StartupParameters": {
"ModelDataDownloadTimeoutInSeconds": 1200,
"ContainerStartupHealthCheckTimeoutInSeconds": 1200,
},
"ComputeResourceRequirements": {
"NumberOfAcceleratorDevicesRequired": 1,
"MinMemoryRequiredInMb": 32768,
},
},
RuntimeConfig={"CopyCount": 1},
)
Repeat for a second, smaller model — a reranker, a classifier, an embedding model — with NumberOfAcceleratorDevicesRequired: 1 and less memory. On a g5.12xlarge you have four A10Gs, so four single-GPU replicas fit per instance in any mix you like. That is the packing win: three models that each needed their own g5.2xlarge (three instances) now share one instance with a spare slot.
Getting ComputeResourceRequirements right is the whole game. Under-request memory and replicas fail to place or OOM under load; over-request and you waste slots. Measure actual GPU memory during a load test (nvidia-smi inside the container, or the GPUMemoryUtilization CloudWatch metric) and add roughly 20 percent headroom rather than guessing.
Step 3: autoscaling, including down to zero
Each component gets its own Application Auto Scaling target. To allow scale to zero, set MinCapacity=0.
aas = boto3.client("application-autoscaling")
rid = "inference-component/llama-31-8b"
aas.register_scalable_target(
ServiceNamespace="sagemaker",
ResourceId=rid,
ScalableDimension="sagemaker:inference-component:DesiredCopyCount",
MinCapacity=0,
MaxCapacity=4,
)
aas.put_scaling_policy(
PolicyName="llama-31-8b-tt",
ServiceNamespace="sagemaker",
ResourceId=rid,
ScalableDimension="sagemaker:inference-component:DesiredCopyCount",
PolicyType="TargetTrackingScaling",
TargetTrackingScalingPolicyConfiguration={
"TargetValue": 5.0,
"PredefinedMetricSpecification": {
"PredefinedMetricType": "SageMakerInferenceComponentConcurrentRequestsPerCopyHighResolution"
},
"ScaleInCooldown": 600,
"ScaleOutCooldown": 60,
},
)
Target tracking will not take you from 1 replica to 0 on its own — you need a step-scaling policy on a "no traffic" alarm to make the last hop.
aas.put_scaling_policy(
PolicyName="llama-31-8b-scale-to-zero",
ServiceNamespace="sagemaker",
ResourceId=rid,
ScalableDimension="sagemaker:inference-component:DesiredCopyCount",
PolicyType="StepScaling",
StepScalingPolicyConfiguration={
"AdjustmentType": "ExactCapacity",
"Cooldown": 600,
"MetricAggregationType": "Maximum",
"StepAdjustments": [{"MetricIntervalUpperBound": 0.0, "ScalingAdjustment": 0}],
},
)
Then a CloudWatch alarm on NoCapacityInvocationFailure (or on Invocations at zero for several consecutive minutes, treat_missing_data="breaching") invokes that policy. AWS's own guidance is to require several consecutive quiet periods — 10 to 15 minutes — so a brief lull does not evict a model you are about to need again.
Scale-out from zero uses the same alarm family in reverse: SageMaker AI emits NoCapacityInvocationFailure when a request arrives for a component at zero replicas, and a step policy on that alarm sets capacity back to 1.
Step 4: invoking a component
The client call names the component, not the endpoint variant:
import json, boto3
rt = boto3.client("sagemaker-runtime")
resp = rt.invoke_endpoint(
EndpointName="shared-gpu",
InferenceComponentName="llama-31-8b",
ContentType="application/json",
Body=json.dumps({"inputs": "Summarize SageMaker inference components in one sentence."}),
)
print(json.loads(resp["Body"].read()))
If the component is at zero replicas, this call fails fast with a capacity error rather than blocking for four minutes. Design for that. The two patterns that work:
- Queue in front. Put an SQS queue or an asynchronous endpoint in front of the cold path; the first request triggers scale-out and the queue absorbs the wait.
- Retry with backoff plus a warm path. Retry the component for a few minutes while serving the request from a cheaper always-warm model (a serverless endpoint or a Bedrock on-demand model). Users get an answer; the GPU still spent the weekend at zero.
Whatever you do, do not put a synchronous, user-facing request path directly onto a scale-to-zero component and hope.
When this is the wrong tool
- Strict p99 latency, 24/7 traffic. Scale to zero saves nothing if you are never idle, and packing adds noisy-neighbour risk. Keep a dedicated endpoint.
- One model, spiky, small. A serverless endpoint is simpler and already scales to zero for you.
- Wildly different accelerator needs. A model needing 8 GPUs and a model needing a CPU slice do not want to share an endpoint's instance type.
- Hard tenant isolation. Regulated multi-tenant setups often need separate endpoints and separate IAM boundaries, whatever the cost model says.
Cost sketch
A dev/staging environment with four models on dedicated ml.g5.2xlarge endpoints runs four instances, 730 hours a month, whether anyone is working or not. The same four models as components on one ml.g5.12xlarge with scale to zero and a 50-hour working week land near a fifth of that, and often less, because staging is idle most of the month. Production usually sees a smaller but still real win from packing alone.
Run the numbers with your own hours before committing — but if your non-production ML environments are running warm GPUs around the clock, this is the cheapest week of engineering you will do this quarter.
Cleanup
Components must go before the endpoint:
for name in ["llama-31-8b", "reranker"]:
sm.delete_inference_component(InferenceComponentName=name)
sm.delete_endpoint(EndpointName="shared-gpu")
sm.delete_endpoint_config(EndpointConfigName="shared-gpu-config")
Also deregister the scalable targets, or stale Application Auto Scaling entries will linger and confuse the next person reading the console.
Checklist
ManagedInstanceScalingenabled withMinInstanceCount: 0.- Least-outstanding-requests routing on the variant.
- Per-component compute requirements measured, not guessed, with ~20 percent memory headroom.
- Target tracking for normal load, step scaling for the 1-to-0 and 0-to-1 hops.
- Cold-start handling in the client: queue, or retry plus a warm fallback.
- Generous startup and health-check timeouts for large artifacts.
- Teardown that removes components and scalable targets, not just the endpoint.
Want a second opinion on your inference bill before you refactor it? We do short endpoint-and-cost reviews that end with a packing plan and a projected monthly number — see our MLOps consulting page, or contact us.