Faster Tokens, Same GPU: Quantization, Speculative Decoding, and Compilation on SageMaker AI

Every team that ships an LLM endpoint eventually asks the same question: can we make this faster without renting a bigger GPU? Usually the answer is yes, and usually it has nothing to do with the instance type. Most self-hosted models on SageMaker AI run in their default precision, with one token generated per forward pass, on a container nobody tuned. Fixing that is often a 2x throughput change with no accuracy story to defend.

This tutorial covers the three optimizations that actually move the needle — quantization, speculative decoding, and compilation — using the SageMaker AI inference optimization toolkit, plus the benchmarking discipline that tells you whether the optimization helped or just moved the bottleneck.

It assumes you can already deploy a real-time endpoint. If not, start with deploying a Hugging Face model to a real-time endpoint and come back.

First: measure the thing you actually care about

Do not start optimizing until you have a baseline, and do not benchmark with a single request. LLM serving has three numbers that matter and they trade off against each other:

• TTFT (time to first token) — dominated by prefill, i.e. prompt length and batch pressure. This is what a chat user feels.
• TPOT / ITL (time per output token, inter-token latency) — dominated by decode. This is what makes streaming feel smooth.
• Throughput (tokens/second across all concurrent requests) — this is what your bill is denominated in.

A change that improves throughput 40% while pushing p99 TTFT from 400 ms to 2.5 s is a regression for a chat product and a win for a batch enrichment job. Decide which you are buying before you tune.

Get a baseline at your real concurrency, with your real prompt shape:

import json, time, statistics, boto3
from concurrent.futures import ThreadPoolExecutor

rt = boto3.client("sagemaker-runtime")
ENDPOINT = "llama-8b-baseline"
PROMPT = "Summarize the following support ticket in two sentences:\n\n" + ("lorem ipsum " * 300)

def one_call():
    body = {"inputs": PROMPT, "parameters": {"max_new_tokens": 256, "temperature": 0.2}}
    t0 = time.perf_counter()
    first = None
    resp = rt.invoke_endpoint_with_response_stream(
        EndpointName=ENDPOINT, ContentType="application/json", Body=json.dumps(body)
    )
    tokens = 0
    for event in resp["Body"]:
        chunk = event.get("PayloadPart", {}).get("Bytes")
        if not chunk:
            continue
        if first is None:
            first = time.perf_counter() - t0
        tokens += 1
    total = time.perf_counter() - t0
    return first, total, tokens

CONCURRENCY, ROUNDS = 16, 10
with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
    results = list(pool.map(lambda _: one_call(), range(CONCURRENCY * ROUNDS)))

ttft = sorted(r[0] for r in results)
print("p50 TTFT", round(ttft[len(ttft) // 2], 3))
print("p99 TTFT", round(ttft[int(len(ttft) * 0.99) - 1], 3))
print("mean tok/s per req", round(statistics.mean(r[2] / r[1] for r in results), 1))

Record the numbers in a file next to your deployment code. Every optimization below gets re-run against the same script at the same concurrency, or you are guessing.

Optimization 1: quantization (almost always first)

Quantization stores weights in fewer bits. It reduces the memory the model occupies, which leaves more HBM for the KV cache, which lets the server hold more concurrent sequences in a batch — and larger batches are where GPU throughput comes from. The latency win is a side effect of the memory win.

Rules of thumb as of 2026:

Scheme	Typical quality cost	Where it fits
FP8 (weights + activations)	negligible on H100/H200-class (p5, p5e)	default choice when the hardware supports it
INT8 / SmoothQuant	small, task dependent	Ampere/Ada GPUs (g5, g6)
AWQ / GPTQ 4-bit	noticeable on reasoning and code tasks	when the model otherwise does not fit, or for cost-first batch work

Run the optimization as a job rather than quantizing inside the container at startup — startup quantization adds minutes to every scale-out event and to every cold start:

from sagemaker.serve.builder.model_builder import ModelBuilder
from sagemaker.serve.builder.schema_builder import SchemaBuilder

builder = ModelBuilder(
    model="meta-textgeneration-llama-3-1-8b-instruct",  # JumpStart model id
    schema_builder=SchemaBuilder("sample input", "sample output"),
    role_arn=role,
    instance_type="ml.g6.2xlarge",
)

optimized = builder.optimize(
    instance_type="ml.g6.2xlarge",
    quantization_config={
        "OverrideEnvironment": {"OPTION_QUANTIZE": "awq"},
    },
    output_path="s3://my-bucket/optimized/llama-3-1-8b-awq/",
    accept_eula=True,
)

predictor = optimized.deploy(
    initial_instance_count=1,
    instance_type="ml.g6.2xlarge",
    endpoint_name="llama-8b-awq",
)

The job writes optimized artifacts to S3 once. Every subsequent deploy and every autoscaling event loads pre-quantized weights, so cold start drops instead of rising.

Optimization 2: speculative decoding (latency at low concurrency)

Decoding is memory-bandwidth bound: generating one token requires streaming the whole model through the GPU. Speculative decoding has a cheap draft produce several candidate tokens, then the target model verifies them in a single forward pass. Accepted tokens are free. Rejected ones cost you the draft work.

Two flavours are available without training anything:

1. Draft-model speculation — a small model from the same family (for example a 1B drafting for an 8B). Best acceptance rates, but the draft occupies GPU memory.
2. N-gram / lookahead speculation — candidates come from repeated spans in the prompt and generation so far. No extra weights. Excellent on summarization, RAG answers with quotation, and code edits, where output overlaps input heavily; nearly useless on free-form creative generation.

Enable it in the same optimize call:

optimized = builder.optimize(
    instance_type="ml.g6.12xlarge",
    speculative_decoding_config={
        "ModelProvider": "SAGEMAKER",   # SageMaker-provided draft model
    },
    accept_eula=True,
)

For a self-supplied draft, point at your own artifacts instead:

speculative_decoding_config={
    "ModelSource": "s3://my-bucket/drafts/llama-3-2-1b/",
    "AcceptEula": True,
}

The number that decides whether this was worth it is the acceptance rate. Log it from the container metrics; below roughly 40% acceptance, speculation typically loses to plain decoding because every rejected draft token is wasted compute. Also expect the win to shrink as concurrency rises — under heavy batching the GPU is already compute-saturated and there is no idle bandwidth for the draft to exploit. Speculative decoding is a low-to-moderate concurrency, latency-sensitive optimization. Measure it at your real load, not at concurrency 1.

Optimization 3: compilation

Compilation (TensorRT-LLM engines on NVIDIA, Neuron compilation for inf2 / trn1) turns the model graph into a hardware-specific engine with fused kernels and pre-planned memory. It buys single-digit-to-30% latency, and it is the least flexible option: engines are pinned to a GPU family, a tensor-parallel degree, and a maximum sequence length. Change the instance type and you recompile.

optimized = builder.optimize(
    instance_type="ml.g6.12xlarge",
    compilation_config={"OverrideEnvironment": {"OPTION_TENSOR_PARALLEL_DEGREE": "4"}},
    accept_eula=True,
)

Do compilation last, once the model, the quantization scheme, and the instance family are settled. If you are considering AWS silicon instead, our tutorial on deploying LLMs to Inferentia2 with Neuron covers that path end to end.

The free tuning nobody does: container settings

Before you reach for anything exotic, check the serving container's own knobs. On the Large Model Inference container these three explain most "why is it slow" tickets:

env = {
    "OPTION_MAX_MODEL_LEN": "8192",          # do not reserve cache for 128k if you send 4k
    "OPTION_MAX_ROLLING_BATCH_SIZE": "32",   # concurrency the server will batch
    "OPTION_GPU_MEMORY_UTILIZATION": "0.92", # how much HBM the KV cache may claim
}

OPTION_MAX_MODEL_LEN is the highest-leverage one. A model advertising a 128k context will pre-plan KV cache for 128k tokens per sequence if you let it, collapsing your achievable batch size. Set it to the longest request you actually serve plus headroom.

Also set a realistic ModelDataDownloadTimeoutInSeconds and ContainerStartupHealthCheckTimeoutInSeconds on the endpoint config for large artifacts — most "deployment failed" reports on 70B-class models are a health check that fired before the weights finished loading.

Prove the quality did not move

Every optimization here is a numerical change to the model. Throughput numbers are meaningless if you silently lost 6 points of accuracy, and "it looked fine in the notebook" is not evidence. Run the same evaluation suite against the baseline and the optimized endpoint and gate on the delta — see automated LLM evaluation gates with fmeval for the harness.

A workable acceptance rule: task metric within 1% absolute of baseline, refusal rate unchanged, and a human spot-check of 50 samples on your two hardest prompt classes. For 4-bit quantization, check long-context and multi-step reasoning prompts specifically; that is where low-bit schemes degrade first and where averaged benchmark scores hide it.

Roll the optimized model out behind canary or shadow traffic rather than swapping it in place. Shadow testing is especially good here: identical production traffic, two numerically different models, real latency comparison, no user exposure.

Decision order

1. Set OPTION_MAX_MODEL_LEN and batch size correctly. Re-benchmark. (Free.)
2. Quantize — FP8 where the hardware allows, INT8/AWQ otherwise. Re-benchmark and re-evaluate.
3. If TTFT/ITL at your real concurrency is still the binding constraint, add speculative decoding and watch the acceptance rate.
4. Freeze the instance family, then compile.
5. Only now consider a bigger or different instance — and compare against scale-to-zero inference components before you commit to always-on capacity.

Teams routinely skip to step 5 and pay for it monthly.

The short version

Optimization on SageMaker AI is a sequence, not a switch. Benchmark at real concurrency with TTFT, inter-token latency, and throughput recorded separately; fix the container's context-length and batch settings first; quantize ahead of time as an optimization job so cold starts get faster instead of slower; add speculative decoding only where acceptance rates justify it; compile last; and gate every step on an evaluation suite so the speed you gained is not accuracy you lost.

Have an endpoint that is too slow or too expensive and no baseline to argue from? Get in touch — our SageMaker consultants will benchmark it, work the optimization order, and hand you the before/after numbers with the quality evidence attached.