Deploy a Hugging Face Model to a SageMaker AI Real-Time Endpoint

Deploying a Hugging Face model to a real-time endpoint is the first thing most teams do on SageMaker AI, and it is also where the first surprise bill comes from. This tutorial walks through the two sane ways to do it in 2026, how to pick an instance, how to add autoscaling, and how to shut it all down properly.

Everything below uses the SageMaker Python SDK v2 (pip install "sagemaker>=2.200") from a Studio space or any machine with AWS credentials and a SageMaker execution role.

Path 1: JumpStart (fastest)

SageMaker JumpStart wraps popular models with tested containers and instance defaults. If the model you want is in the catalog, this is three lines:

from sagemaker.jumpstart.model import JumpStartModel

model = JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct")
predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.2xlarge",
    endpoint_name="mistral-7b-instruct",
)

You can list catalog IDs with sagemaker.jumpstart.notebook_utils.list_jumpstart_models(). Some models require you to accept a EULA; the SDK will tell you which argument to pass. JumpStart chooses the serving container for you (usually the Large Model Inference container for LLMs), which is the main reason to use it: the container configuration is where most hand-rolled deployments go wrong.

Path 2: HuggingFaceModel (any Hub model)

For a model that is not in JumpStart, or when you want control over the container and its environment, use the Hugging Face integration directly. This example deploys a sentence-embedding model on a CPU instance, which is a common and cheap real-time workload:

import sagemaker
from sagemaker.huggingface import HuggingFaceModel

role = sagemaker.get_execution_role()

hub = {
    "HF_MODEL_ID": "sentence-transformers/all-MiniLM-L6-v2",
    "HF_TASK": "feature-extraction",
}

model = HuggingFaceModel(
    env=hub,
    role=role,
    transformers_version="4.49",
    pytorch_version="2.6",
    py_version="py312",
)

predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.c7i.xlarge",
    endpoint_name="minilm-embeddings",
)

The transformers_version / pytorch_version / py_version triple must match a published Hugging Face Deep Learning Container; the current supported combinations are listed in the Hugging Face on SageMaker docs. If you get an "unsupported version" error, that table is the fix.

For LLM text generation, swap the container for the TGI image instead of the default inference toolkit:

from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

llm_model = HuggingFaceModel(
    role=role,
    image_uri=get_huggingface_llm_image_uri("huggingface"),
    env={
        "HF_MODEL_ID": "meta-llama/Llama-3.1-8B-Instruct",
        "HF_TOKEN": "<your Hugging Face token>",
        "SM_NUM_GPUS": "1",
        "MAX_INPUT_TOKENS": "4096",
        "MAX_TOTAL_TOKENS": "8192",
    },
)

llm = llm_model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.2xlarge",
    endpoint_name="llama-3-1-8b",
    container_startup_health_check_timeout=600,
)

Gated models need HF_TOKEN; in production put it in Secrets Manager and inject it rather than pasting it into a notebook. The long health-check timeout matters because an 8B model takes several minutes to download and load.

Choosing an instance

The decision is about accelerator memory first and cost second.

Workload	Start with	Why
Embeddings, classifiers, small encoders	ml.c7i.xlarge / ml.m7i.large	CPU is fine; GPUs sit idle
7B to 8B LLM, FP16	ml.g5.2xlarge (24 GB A10G)	Fits with room for KV cache
7B to 8B LLM, higher throughput	ml.g6.2xlarge / ml.g6e.2xlarge (L4 / L40S)	Newer GPUs, better price-performance
13B to 70B LLM	ml.g5.12xlarge and up, or ml.p4d	Tensor parallel across GPUs
Steady high-volume LLM traffic	ml.inf2.xlarge and up (Inferentia2)	Lowest cost per token once you have compiled the model

Rule of thumb for FP16 weights: parameters x 2 bytes, plus 20 to 40 percent for the KV cache at your context length. An 8B model is about 16 GB of weights, so a 24 GB GPU works; a 70B model does not fit on anything smaller than four 80 GB GPUs without quantization. Check the SageMaker AI pricing page for the current per-hour rates before committing; the gap between g5 and g6e on the same workload is often the whole margin.

Invoking the endpoint

Through the predictor object:

from sagemaker.serializers import JSONSerializer
from sagemaker.deserializers import JSONDeserializer

predictor.serializer = JSONSerializer()
predictor.deserializer = JSONDeserializer()

print(predictor.predict({"inputs": "SageMaker endpoints are"}))

From an application that does not have the SageMaker SDK installed, use the runtime client in boto3:

import boto3, json

runtime = boto3.client("sagemaker-runtime")
response = runtime.invoke_endpoint(
    EndpointName="llama-3-1-8b",
    ContentType="application/json",
    Body=json.dumps({
        "inputs": "Explain autoscaling in one sentence.",
        "parameters": {"max_new_tokens": 64, "temperature": 0.2},
    }),
)
print(json.loads(response["Body"].read()))

For streaming token output, invoke_endpoint_with_response_stream returns chunks as the model generates them; TGI and LMI both support it.

Autoscaling

A single instance with no scaling policy is either over-provisioned or a single point of failure. Register the endpoint variant with Application Auto Scaling and scale on invocations per instance:

import boto3

asg = boto3.client("application-autoscaling")
resource_id = "endpoint/llama-3-1-8b/variant/AllTraffic"

asg.register_scalable_target(
    ServiceNamespace="sagemaker",
    ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    MinCapacity=1,
    MaxCapacity=4,
)

asg.put_scaling_policy(
    PolicyName="invocations-per-instance",
    ServiceNamespace="sagemaker",
    ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 20.0,
        "PredefinedMetricSpecification": {
            "PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"
        },
        "ScaleInCooldown": 600,
        "ScaleOutCooldown": 120,
    },
)

Pick TargetValue from a load test, not a guess: run a fixed concurrency against one instance, note invocations per minute at the latency you can accept, and set the target a little below it. For LLMs, scaling on SageMakerVariantInvocationsPerInstance is crude because requests vary in token count; a custom CloudWatch metric on queue depth or p95 latency is better once you are past the first deployment. Scale-out on a GPU instance takes several minutes because the model has to load, so keep MinCapacity high enough to absorb a burst while new instances come up.

If traffic is spiky and latency-tolerant, consider an asynchronous endpoint with scale-to-zero instead; see our guide to choosing an inference type.

Teardown and cost hygiene

An endpoint bills per instance-hour whether or not it receives traffic. The deploy() call creates three resources: a model, an endpoint configuration, and the endpoint. Delete all three:

predictor.delete_endpoint(delete_endpoint_config=True)
predictor.delete_model()

Habits that save real money:

• Tag every endpoint with an owner and an expiry and run a nightly job that reports (or deletes) endpoints past their date.
• Set a CloudWatch alarm on Invocations being zero for 24 hours.
• Use a scaling policy with MinCapacity=1 rather than a static two-instance deployment for development.
• Keep experimentation on serverless endpoints where the model fits, so idle time costs nothing.

Where to go next

Once an endpoint is live, the next questions are monitoring (data capture and Model Monitor), deploying through a Model Registry approval rather than a notebook, and deciding whether real-time is even the right inference type. We cover the registry path in A Minimal MLOps Loop.

Need help sizing or hardening an LLM endpoint? Talk to us.