NeuralArmada | Benchmark Before You Scale: Load-Testing and Right-Sizing SageMaker AI LLM Endpoints

Every SageMaker AI endpoint right-sizing argument we walk into sounds the same: someone picked ml.g5.2xlarge because a blog post said so, traffic grew, latency got ugly, and the fix proposed is "a bigger instance." Almost always the real answer comes from thirty minutes of load testing. This tutorial shows how to benchmark an LLM endpoint properly — the metrics that matter, a concurrency sweep you can run from a laptop, how to read the CloudWatch side, and how to turn the results into a cost-per-million-tokens number you can defend in a budget meeting.

Assumes you already have an endpoint live (see Deploy a Hugging Face Model to a SageMaker AI Real-Time Endpoint) and credentials with sagemaker:InvokeEndpoint plus cloudwatch:GetMetricData.

Step 1: measure the right four numbers

Average latency is useless for generative workloads because response time scales with output length. Benchmark these instead:

TTFT (time to first token) — what a user perceives as "it's thinking." Driven by prefill, queueing, and input length.
ITL (inter-token latency), or its inverse, tokens per second per request — what a user perceives as "it's typing."
Throughput — total output tokens per second across all concurrent requests. This is the number that sets your cost.
Error and queue behaviour — 5xx rate, and SageMaker's ModelLatency vs OverheadLatency split.

TTFT and throughput trade against each other. One request on an idle GPU gets the best TTFT and terrible throughput; thirty concurrent requests saturate the batch and invert that. Your job is to find the concurrency where throughput is high and TTFT is still inside your SLO.

Step 2: build a realistic payload set

Benchmarks run with 20-token prompts lie. Prefill cost is roughly linear in input length, and KV cache pressure — which decides how many requests the server can batch — depends on input plus output length. Sample real traffic if you have it; otherwise fix a representative shape, for example 2,000 input tokens and 256 output tokens for a RAG workload, or 400/800 for a chat assistant.

import json, random
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
corpus = open("sample_docs.txt").read().split()

def make_payload(input_tokens=2000, output_tokens=256):
    words = " ".join(random.choices(corpus, k=input_tokens))
    return {
        "inputs": f"Answer using the context.\n\nContext: {words}\n\nQuestion: What is the main risk?",
        "parameters": {
            "max_new_tokens": output_tokens,
            "min_new_tokens": output_tokens,   # pin output length so runs are comparable
            "temperature": 0.2,
            "details": True,
        },
    }

Pinning min_new_tokens to max_new_tokens is the trick that makes runs comparable across concurrency levels. Without it, a model that happens to stop early looks faster.

Step 3: a streaming load generator

Use invoke_endpoint_with_response_stream so you can time the first token instead of only the whole response.

import asyncio, json, time, statistics
import boto3

runtime = boto3.client("sagemaker-runtime")
ENDPOINT = "llama-3-1-8b"

def one_request(payload):
    start = time.perf_counter()
    first = None
    tokens = 0
    resp = runtime.invoke_endpoint_with_response_stream(
        EndpointName=ENDPOINT,
        ContentType="application/json",
        Body=json.dumps(payload),
    )
    for event in resp["Body"]:
        chunk = event.get("PayloadPart", {}).get("Bytes")
        if not chunk:
            continue
        if first is None:
            first = time.perf_counter() - start
        tokens += 1
    total = time.perf_counter() - start
    return {"ttft": first, "total": total, "tokens": tokens}

async def run(concurrency, requests_per_worker, payload_fn):
    loop = asyncio.get_running_loop()
    async def worker():
        out = []
        for _ in range(requests_per_worker):
            out.append(await loop.run_in_executor(None, one_request, payload_fn()))
        return out
    t0 = time.perf_counter()
    results = [r for rs in await asyncio.gather(*[worker() for _ in range(concurrency)]) for r in rs]
    wall = time.perf_counter() - t0
    return results, wall

Chunk counting is an approximation of token counting — most LMI and TGI stream one token per chunk, but verify against the details field in the final chunk before you quote the number to anyone.

Step 4: sweep concurrency

def pct(values, p):
    values = sorted(values)
    return values[int(len(values) * p) - 1]

for c in [1, 2, 4, 8, 16, 32, 64]:
    results, wall = asyncio.run(run(c, 5, make_payload))
    out_tokens = sum(r["tokens"] for r in results)
    print(
        f"concurrency={c:>3} "
        f"ttft_p50={pct([r['ttft'] for r in results], .50):.2f}s "
        f"ttft_p95={pct([r['ttft'] for r in results], .95):.2f}s "
        f"e2e_p95={pct([r['total'] for r in results], .95):.2f}s "
        f"throughput={out_tokens / wall:.1f} tok/s"
    )

Warm the endpoint with a few throwaway requests first — the first invocation after deployment pays for CUDA graph capture and cache warmup. Pin MinCapacity = MaxCapacity = 1 on the autoscaling target for the duration of the test, or you will be benchmarking the scaling policy rather than the instance.

A typical result on one A10G with an 8B model at 2,000/256 looks roughly like this:

Concurrency	TTFT p95	Throughput	Verdict
1	0.4 s	38 tok/s	GPU mostly idle
8	0.9 s	210 tok/s	healthy
16	1.6 s	290 tok/s	knee of the curve
32	4.8 s	310 tok/s	queueing, not computing
64	12 s+	305 tok/s	requests waiting in line

Throughput flattening while TTFT climbs is the signature of a saturated continuous batch. Past that point, more traffic needs another instance, not patience.

Step 5: cross-check against CloudWatch

Client-side numbers include your own network. SageMaker publishes the server's view in the AWS/SageMaker namespace: ModelLatency, OverheadLatency, Invocations, InvocationsPerInstance, and Invocation5XXErrors. Instance-level GPUUtilization, GPUMemoryUtilization, and CPUUtilization live in /aws/sagemaker/Endpoints.

import datetime as dt
cw = boto3.client("cloudwatch")
end = dt.datetime.utcnow(); start = end - dt.timedelta(minutes=20)

resp = cw.get_metric_data(
    MetricDataQueries=[{
        "Id": "gpu",
        "MetricStat": {
            "Metric": {
                "Namespace": "/aws/sagemaker/Endpoints",
                "MetricName": "GPUUtilization",
                "Dimensions": [
                    {"Name": "EndpointName", "Value": ENDPOINT},
                    {"Name": "VariantName", "Value": "AllTraffic"},
                ],
            },
            "Period": 60, "Stat": "Average",
        },
    }],
    StartTime=start, EndTime=end,
)
print(resp["MetricDataResults"][0]["Values"])

Read it this way:

GPU utilization low, latency high — you are bottlenecked on tokenization, payload size, or client concurrency, not the accelerator. Check OverheadLatency.
GPU memory near 100 percent at modest concurrency — KV cache is the limit. Shorten max context, quantize the KV cache, or move to a bigger-memory GPU.
OverheadLatency a large share of total — serialization and routing overhead; large JSON payloads and base64 images are usual suspects.
5XX climbing at high concurrency — the server is rejecting past its queue limit. Tune the container's max concurrent requests rather than letting clients retry blindly.

Step 6: convert to cost per million tokens

This is the number that settles instance arguments.

cost per 1M output tokens = (instance $/hour / sustained tokens per second) * 1e6 / 3600

At the knee from the table above — 290 tok/s on an ml.g5.2xlarge — and a list price near $1.52/hour, that is roughly $1.46 per million output tokens at full utilization. Now run the same test on ml.g6e.2xlarge and on ml.inf2.xlarge, and compare the same number. The winner is frequently not the cheapest per hour. Two caveats people forget: real endpoints never run at 100 percent utilization, so divide throughput by your realistic duty cycle (0.3 is common), and compare the result against Bedrock's published per-token price before you commit to self-hosting at all — see Bedrock or SageMaker AI for Generative AI?.

Step 7: feed the results back

Set the autoscaling target from the knee, not from a round number. If 16 concurrent requests per instance is the limit, and average request duration is 8 seconds, that is about 120 invocations per minute per instance — set SageMakerVariantInvocationsPerInstance a little below it.
Keep headroom for scale-out time. A GPU instance takes minutes to load a model, so the policy must trigger before you are at the cliff.
Rerun on every model or container change. A vLLM or LMI version bump can move throughput 30 percent in either direction.
Store the results next to the model version — a benchmark in the Model Registry's metadata, or as an evaluation artifact, keeps the next engineer from relitigating the instance choice. If you are already optimizing the model itself, pair this with Faster Tokens, Same GPU.
Automate it. A nightly benchmark job against a canary endpoint turns performance regressions into a failed build rather than a customer complaint.

Common mistakes

Benchmarking from a laptop over home Wi-Fi, then blaming the GPU. Run the load generator on EC2 in the same region.
Leaving autoscaling on during the test.
Measuring with 50-token prompts and deploying 4,000-token RAG traffic.
Reporting averages instead of p95.
Forgetting to delete the endpoint afterwards. Benchmark instances are usually the largest ones you will ever launch.

Right-sizing is one of the cheapest wins in an ML platform: the same traffic on the correct instance, at the correct autoscaling target, routinely costs half as much with better tail latency. If you want a second pair of eyes on an endpoint's capacity plan or its benchmark numbers, get in touch.