NeuralArmada | Choosing an Inference Type: Real-Time vs Serverless vs Async vs Batch

SageMaker AI offers four ways to run inference, and picking the wrong one is the most common cost mistake we see in client accounts: a real-time GPU endpoint serving a nightly scoring job, or a batch transform job bolted onto an API that needed answers in 200 ms. This guide lays out the four options, the decision table we use, the cost math per traffic shape, and the code for each.

The four options in one table

	Real-time	Serverless	Asynchronous	Batch transform
Latency	Milliseconds	Milliseconds, plus cold starts	Seconds to minutes	Minutes to hours (job)
Payload limit	6 MB, 60 s timeout	6 MB, 60 s timeout	1 GB, up to 1 hour per request	Whole S3 datasets
Scale to zero	No (min 1 instance)	Yes	Yes	N/A (job-based)
GPU support	Yes	No	Yes	Yes
Billing	Per instance-hour	Per ms of compute + requests	Per instance-hour (zero when scaled in)	Per instance-hour for the job
Best for	User-facing APIs, steady traffic	Spiky, low-volume CPU models	Large payloads, long inference, bursty GPU work	Scoring datasets on a schedule

Check the deployment guide for current limits; the payload and timeout numbers above have been stable for years but are the kind of thing AWS raises quietly.

The decision in three questions

Does a human or a synchronous system wait for the answer? If no, stop: use batch transform for datasets or asynchronous inference for individual requests. Most "we need real-time" requirements evaporate here.
Does the model need a GPU? If yes, serverless is out. Real-time or asynchronous, depending on how bursty the traffic is.
What does the traffic look like over a day? Steady or predictable: real-time with autoscaling. Mostly idle with occasional bursts: serverless (CPU) or asynchronous with scale-to-zero (GPU).

Cost math per traffic shape

Assume a CPU model that takes 100 ms per request and fits comfortably on an ml.m5.large. Use your region's rates from the pricing page; the shapes below are what matter.

Shape A: 10 requests per second, all day. 864,000 requests per day. One real-time instance handles this with headroom; you pay 24 instance-hours per day regardless. Serverless would bill 864,000 x 100 ms = 24 compute-hours of memory-time plus a per-request charge, which works out several times more expensive than the instance. Real-time wins.

Shape B: 2,000 requests per day, clustered in office hours. A real-time instance still costs 24 instance-hours for 200 seconds of actual work. Serverless bills about 200 seconds of compute plus 2,000 requests, which is pocket change. Serverless wins by an order of magnitude, and cold starts (a second or two for a small CPU model) are tolerable.

Shape C: 50,000 documents scored every night. Batch transform spins up two ml.m5.xlarge instances for 20 minutes and stops. That is under one instance-hour per day. A real-time endpoint kept alive for the same job is 24. Batch wins.

Shape D: a 2B-parameter vision model, 300 requests per hour in bursts, 4-second inference. GPU rules out serverless. Real-time on ml.g5.xlarge is 24 GPU-hours per day for 20 minutes of work. Asynchronous inference with a scale-to-zero policy runs the instance only while the queue is non-empty: perhaps two or three GPU-hours a day. Asynchronous wins, at the cost of a few minutes of start-up latency when scaling from zero.

The pattern: real-time is the right answer for steady traffic and the wrong answer for everything else.

Code: real-time

from sagemaker.model import Model

model = Model(image_uri=image_uri, model_data=model_data, role=role)
predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.m5.large",
    endpoint_name="scorer-realtime",
)
print(predictor.predict(payload))

Add a target-tracking autoscaling policy before sending production traffic; we cover the exact configuration in our real-time endpoint tutorial.

Code: serverless

from sagemaker.serverless import ServerlessInferenceConfig

serverless_config = ServerlessInferenceConfig(
    memory_size_in_mb=4096,     # 1024 to 6144, in 1 GB steps
    max_concurrency=20,
)
predictor = model.deploy(
    serverless_inference_config=serverless_config,
    endpoint_name="scorer-serverless",
)

Memory size also sets the CPU allocation, so a model that is slow at 2 GB often speeds up at 4 GB for less money per request. Provisioned concurrency is available if cold starts are a problem; it reintroduces a fixed cost, so only enable it for the concurrency you actually need. Details are in the serverless endpoints docs.

Code: asynchronous

from sagemaker.async_inference import AsyncInferenceConfig

async_config = AsyncInferenceConfig(
    output_path=f"s3://{bucket}/async-output/",
    max_concurrent_invocations_per_instance=4,
    notification_config={
        "SuccessTopic": success_topic_arn,
        "ErrorTopic": error_topic_arn,
    },
)
async_predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.xlarge",
    async_inference_config=async_config,
    endpoint_name="vision-async",
)

# Invoke: the input is uploaded to S3 and the call returns immediately.
response = async_predictor.predict_async(input_path=f"s3://{bucket}/async-input/doc-001.json")
print(response.output_path)   # poll this, or subscribe to the SNS topic

Scale-to-zero is a scaling policy on the ApproximateBacklogSizePerInstance metric with MinCapacity=0:

asg.register_scalable_target(
    ServiceNamespace="sagemaker",
    ResourceId="endpoint/vision-async/variant/AllTraffic",
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    MinCapacity=0, MaxCapacity=4,
)
asg.put_scaling_policy(
    PolicyName="backlog",
    ServiceNamespace="sagemaker",
    ResourceId="endpoint/vision-async/variant/AllTraffic",
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 5.0,
        "CustomizedMetricSpecification": {
            "MetricName": "ApproximateBacklogSizePerInstance",
            "Namespace": "AWS/SageMaker",
            "Dimensions": [{"Name": "EndpointName", "Value": "vision-async"}],
            "Statistic": "Average",
        },
        "ScaleInCooldown": 300, "ScaleOutCooldown": 60,
    },
)

See the asynchronous inference docs for the HasBacklogWithoutCapacity step-scaling policy that wakes an endpoint from zero more reliably than target tracking alone.

Code: batch transform

transformer = model.transformer(
    instance_count=2,
    instance_type="ml.m5.xlarge",
    strategy="MultiRecord",
    assemble_with="Line",
    output_path=f"s3://{bucket}/batch-output/",
    max_payload=6,
)
transformer.transform(
    data=f"s3://{bucket}/batch-input/",
    content_type="text/csv",
    split_type="Line",
    join_source="Input",    # write input columns next to predictions
    wait=True,
)

join_source="Input" is the option people miss: it keeps the identifier columns alongside each prediction so you do not have to re-join by row order. Schedule the job with EventBridge Scheduler or as a step in a SageMaker Pipeline. Full options are in the batch transform docs.

Combining them

The same model artifact can sit behind more than one of these at once. A common production layout is real-time for the interactive path, asynchronous for the "upload a large file" path, and batch transform for the nightly backfill, all from one Model Registry package. Deciding that layout up front, and putting it behind the same API, is usually a one-day design exercise that saves far more than a day of compute every month.

Want someone to look at your endpoint bill and tell you which of these you should be using? Contact us.