Most teams reach for quantization first when an LLM bill gets uncomfortable, and that is the right first move — it is a deployment-time change with no training loop attached. But quantization has a floor. Once you are running an 8B model at INT8 on the smallest GPU that fits, the only remaining lever is to stop using a general-purpose model for a narrow job.
That is what distillation does. You take the large model that already behaves the way you want, use it to teach a much smaller model your specific task, and then serve the small one. For the classification, extraction, routing, and summarization workloads that make up most production LLM traffic, a well-distilled 1B–3B student regularly matches a 70B teacher on the only metric that matters — your task rubric — at a fraction of the cost per token.
This tutorial walks the full loop on SageMaker AI: choosing a task that is worth distilling, generating teacher data, fine-tuning the student, gating the swap on evaluation, and deploying it. It assumes you have read Fine-Tune Llama 3.x on SageMaker AI with JumpStart and have an execution role with S3 and SageMaker permissions.
Decide whether the task is distillable
Distillation is not a general-purpose cost lever. It works when the task is narrow, the output is structured or short, and the distribution of inputs is stable. Run this checklist before you spend a sprint:
- Is the output space small or schematic? Labels, JSON payloads, routing decisions, short extractive summaries — good. Open-ended reasoning, long-form drafting, or anything where users will ask arbitrary questions — bad.
- Do you have real production inputs? You need thousands of them. Distilling on synthetic inputs you invented yourself teaches the student your imagination, not your traffic.
- Is the teacher actually good at the task? Distillation copies behaviour, including mistakes. If the teacher is at 82% on your rubric, the student's ceiling is roughly 82%.
- Is the volume high enough to pay back? The loop below costs a few GPU-days plus teacher inference. At 50,000 requests a month, the savings are real within weeks. At 500, leave it alone.
If the answer to the last one is no, the cheaper win is usually inference components and scale-to-zero, not a training project.
Step 1: Freeze an evaluation set before you generate anything
The single most common failure in a distillation project is that the team builds a student, likes the vibes, ships it, and discovers three weeks later that a category of input regressed badly. Build the gate first.
Pull a stratified sample of 300–800 real production inputs, stratified by whatever segments matter (customer tier, document type, language, length). Label them with your best available source of truth — human review for the hard slices, teacher output plus spot-checking for the easy ones. Store it as JSONL in S3, versioned, and never train on it.
{"id":"ev-0001","segment":"invoice","input":"...","expected":{"vendor":"Acme Ltd","total":"1420.00","currency":"GBP"}}
{"id":"ev-0002","segment":"receipt","input":"...","expected":{"vendor":"Bl\u00fcm GmbH","total":"38.90","currency":"EUR"}}
Score both the teacher and your current production setup against it now, so you have the two numbers the student has to beat or match. The harness in Evaluate Before You Ship is exactly the right shape for this.
Step 2: Generate teacher labels with batch transform, not a live endpoint
Teacher inference over 30,000–100,000 examples is a batch job. Running it against a real-time endpoint is slow, fragile, and about three times the price of the batch equivalent. Deploy nothing; submit a batch transform against the teacher model package.
from sagemaker.jumpstart.model import JumpStartModel
teacher = JumpStartModel(model_id="meta-textgeneration-llama-3-3-70b-instruct")
transformer = teacher.transformer(
instance_count=4,
instance_type="ml.g5.48xlarge",
strategy="SingleRecord",
assemble_with="Line",
output_path="s3://my-bucket/distill/teacher-out/",
max_payload=1,
)
transformer.transform(
data="s3://my-bucket/distill/prompts/",
content_type="application/jsonlines",
split_type="Line",
job_name="teacher-label-run-07",
)
transformer.wait()
Two things that save real money here:
- Ask the teacher for the final answer only. If you want the student to reason, ask the teacher to produce a short rationale and the answer, then decide deliberately whether the student is trained on both (rationale distillation, better generalization, more output tokens) or on the answer alone (cheaper, more brittle on unseen phrasings).
- Use a strict output schema and validate every line. Parse each teacher output as you write it; drop anything that does not validate. A silent 4% of malformed JSON in the training set shows up later as a student that occasionally emits malformed JSON.
import json, pathlib
kept, dropped = [], 0
for line in pathlib.Path("teacher-out.jsonl").read_text().splitlines():
rec = json.loads(line)
try:
payload = json.loads(rec["generated_text"])
assert {"vendor", "total", "currency"} <= payload.keys()
except Exception:
dropped += 1
continue
kept.append({"prompt": rec["prompt"], "completion": json.dumps(payload, sort_keys=True)})
print(f"kept={len(kept)} dropped={dropped}")
If dropped is above about 5%, fix the teacher prompt before you fix anything else.
Step 3: Curate, do not just dump
Raw teacher output is not a training set. Three cheap passes lift student quality more than any hyperparameter you will tune:
- Deduplicate near-identical prompts. Production traffic is heavily repetitive; 40,000 rows often contain 12,000 distinct situations. MinHash or embedding-based dedup at 0.95 similarity is enough.
- Rebalance the tail. If 70% of your traffic is one document type, the student will be excellent at it and useless elsewhere. Upsample rare segments to at least a few hundred examples each.
- Filter with a verifier where one exists. For extraction, check numbers against the source text. For classification, drop examples where the teacher's self-consistency across two samples disagrees. Verifiable filtering is the same idea used in reinforcement fine-tuning with GRPO, applied to a cheaper part of the pipeline.
Hold out 5% of the curated set as a training-time validation split. This is not your evaluation set from Step 1 — that one stays sealed.
Step 4: Fine-tune the student
LoRA on a 1B–8B instruct model is the default. Full fine-tuning buys a little more on very large datasets and costs far more to run and to store.
from sagemaker.jumpstart.estimator import JumpStartEstimator
student = JumpStartEstimator(
model_id="meta-textgeneration-llama-3-2-3b-instruct",
instance_type="ml.g5.12xlarge",
environment={"accept_eula": "true"},
use_spot_instances=True,
max_wait=36000,
checkpoint_s3_uri="s3://my-bucket/distill/ckpt/",
)
student.set_hyperparameters(
instruction_tuned="True",
epoch="3",
learning_rate="1e-4",
per_device_train_batch_size="4",
peft_type="lora",
lora_r="16",
lora_alpha="32",
lora_dropout="0.05",
max_input_length="2048",
)
student.fit({"training": "s3://my-bucket/distill/curated/"})
Practical notes:
- Three epochs is a starting point, not a rule. Watch the validation loss; on a tightly-scoped extraction task, two epochs is often the sweet spot and three starts memorizing vendor names.
- Use managed spot with checkpointing. These runs are interruption-tolerant and typically 60–70% cheaper. The pattern is the same one described in Multi-Node FSDP Training.
- Log every run to MLflow. You will train six or eight students before one passes the gate, and in three months nobody will remember which S3 prefix corresponded to which dataset revision. Managed MLflow on SageMaker AI exists for exactly this.
Step 5: Gate the swap on evaluation, then shadow it
Score the student against the sealed evaluation set, per segment. A single aggregate number hides the failure mode that matters: students usually lose on the rare segments, not on the average.
Set the promotion rule numerically and in advance. Something like: overall rubric score within 2 points of teacher, no segment more than 5 points below teacher, malformed-output rate under 0.5%, p95 latency under 400 ms. Wire that as a condition step in the pipeline so a failing student cannot be registered as Approved — see A Minimal MLOps Loop and Prove It: Bias Reports, Model Cards, and Approval Gates for the registry mechanics.
Then, before the student takes real traffic, run it as a shadow variant behind the teacher endpoint for a few days. Shadow testing compares the two on live production inputs — including the weird ones your evaluation set never contained — with no user impact. The configuration is in Zero-Downtime Endpoint Updates.
Step 6: Serve the student properly
A 3B student on ml.g5.xlarge behaves very differently from a 70B teacher on four ml.g5.48xlarge nodes, and the deployment should change accordingly:
- Right-size down aggressively. Re-run the instance comparison from scratch; the student may fit comfortably on a single small GPU, or on Inferentia2 for another large step down in cost per token (Cheaper Tokens on Purpose).
- If you distilled several tasks, serve them as adapters. One base model, many LoRA adapters on one GPU is usually cheaper than one endpoint per task — see Multi-Adapter LoRA Inference.
- Keep the teacher reachable. Route low-confidence or schema-invalid student outputs back to the teacher. A 3% fallback rate preserves quality and still leaves you with 97% of the traffic on the cheap path.
- Monitor for the drift that distillation causes. The student's competence is frozen at the teacher's behaviour on last quarter's inputs. Track malformed-output rate and per-segment agreement with periodic teacher sampling, using the alarm plumbing from Detect Drift in Production.
What this typically costs and saves
A realistic engagement of this shape runs about two to three weeks: two days to build and label the evaluation set, two days for teacher generation and curation, three to four days of student training iterations, and a week of gating, shadow testing, and rollout. Compute is usually a few hundred to low thousands of dollars, dominated by teacher batch inference.
The payoff is structural rather than marginal. Moving a high-volume extraction workload from a 70B teacher to a 3B student on a single small GPU commonly drops serving cost by 10–20x and cuts p95 latency by more than half, which frequently changes what the product is allowed to do — synchronous instead of queued, per-document instead of per-batch.
The two rules worth repeating: seal the evaluation set before you generate a single teacher label, and never ship a student that has not run in shadow. Everything else in the loop is recoverable.
Considering distilling a production LLM workload, or want a second opinion on whether your task is a good candidate before committing a sprint? Talk to us.