Every team we onboard has a deployment story and almost none of them have an evaluation story. The model gets fine-tuned, someone eyeballs twenty generations in a notebook, a colleague says "yeah that looks better", and it ships. That works exactly until the second fine-tune, when nobody can say whether the new checkpoint is actually better than the one already serving traffic.
This tutorial builds the missing piece: a repeatable, automated evaluation step for generative models on SageMaker AI. We use fmeval (the open-source Foundation Model Evaluations library AWS maintains) for the metrics, run it against a live SageMaker AI endpoint, log the results to managed MLflow, and then wire the whole thing into a SageMaker Pipeline as a quality gate that refuses to register a model that regressed.
Everything below runs from a Studio space or any machine with AWS credentials and a SageMaker execution role, using sagemaker>=2.200, fmeval>=1.2, and mlflow>=2.16 with sagemaker-mlflow.
What "evaluation" actually means for a generative model
Classic ML evaluation is a single number against a held-out set. Generative evaluation is not; it is a small portfolio of different question types, and mixing them up is the most common mistake we see:
| Dimension | Question it answers | Typical metric |
|---|---|---|
| Accuracy / task quality | Does the output match the reference? | Exact match, F1, ROUGE, METEOR, BERTScore |
| Factual knowledge | Does it know the domain? | Factual knowledge score on a QA set |
| Summarization quality | Is the summary faithful and useful? | ROUGE-L, BERTScore |
| Robustness | Does a typo or button-mash change the answer? | Semantic robustness (delta under perturbation) |
| Toxicity | Will it say something that ends up in a screenshot? | Detoxify / toxicity classifier scores |
| Prompt stereotyping / bias | Does it prefer one demographic framing? | Stereotyping score |
| Cost & latency | Can you afford to serve it? | p95 latency, tokens/sec, $/1k requests |
fmeval ships built-in algorithms for the first six. The last row is yours — measure it in the same job, because a checkpoint that is two points better on ROUGE and 3x slower is usually a worse product.
Step 1: A curated evaluation dataset beats a benchmark
Public benchmarks tell you whether a model is good in general. They tell you nothing about whether it is good at your task. Build a JSON Lines file of 100–500 examples from your own traffic, with a reference answer per row:
{"question": "What is the refund window for a damaged item?", "answer": "30 days from delivery.", "context": "Damaged goods may be returned within 30 days of delivery..."}
{"question": "Can I change the shipping address after dispatch?", "answer": "No, once dispatched the address is fixed.", "context": "Orders that have been dispatched cannot be redirected..."}
Upload it to S3 and describe it to fmeval with a DataConfig. The *_location fields are JMESPath expressions into each record, which is what lets you point the library at data you already have instead of reshaping it:
from fmeval.data_loaders.data_config import DataConfig
from fmeval.constants import MIME_TYPE_JSONLINES
data_config = DataConfig(
dataset_name="support_qa_v3",
dataset_uri="s3://neuralarmada-evals/support_qa_v3.jsonl",
dataset_mime_type=MIME_TYPE_JSONLINES,
model_input_location="question",
target_output_location="answer",
)
Two rules that save pain later. Version the file (_v3 in the name, never overwrite) so that a metric change always means a model change, not a data change. And keep it out of your training set — a fine-tune that has seen the eval rows will report numbers you cannot ship on.
Step 2: Point fmeval at a SageMaker AI endpoint
fmeval talks to a model through a ModelRunner. For a real-time SageMaker AI endpoint there is one built in; you tell it how to build the request and where the generated text lives in the response:
from fmeval.model_runners.sm_model_runner import SageMakerModelRunner
model_runner = SageMakerModelRunner(
endpoint_name="llama-3-1-8b-instruct",
content_template='{"inputs": $prompt, "parameters": {"max_new_tokens": 256, "temperature": 0.0, "do_sample": false}}',
output='[0].generated_text',
content_type="application/json",
accept_type="application/json",
)
Three details matter here:
$promptis substituted with the JSON-escaped model input. If your container expects the OpenAI-stylemessagesarray, put that shape incontent_templateinstead.outputis a JMESPath into the response body. Get it wrong and every metric comes back near zero — before debugging your model, alwaysprint(model_runner.predict("Say hello."))once.- Set
temperature=0anddo_sample=false. Sampling turns your evaluation into a random number generator; you want run-to-run differences to come from the model, not the dice.
For a model that is not deployed yet, use JumpStartModelRunner, or evaluate against a Bedrock model with BedrockModelRunner — useful when the real question is "is our fine-tune actually beating the managed API we could just call?"
Step 3: Run the evaluations
Each algorithm is a class with a config and an evaluate() method. Start with accuracy on your task:
from fmeval.eval_algorithms.qa_accuracy import QAAccuracy, QAAccuracyConfig
qa = QAAccuracy(QAAccuracyConfig(target_output_delimiter="<OR>"))
qa_results = qa.evaluate(
model=model_runner,
dataset_config=data_config,
prompt_template="Answer using only the context provided.\n\nQuestion: $model_input\n\nAnswer:",
save=True, # writes per-record outputs to /tmp/eval_results
num_records=300,
)
for r in qa_results:
for score in r.dataset_scores:
print(f"{r.eval_name}: {score.name} = {score.value:.4f}")
Then robustness and toxicity, which need no reference answers and so can run on raw production prompts:
from fmeval.eval_algorithms.general_semantic_robustness import (
GeneralSemanticRobustness, GeneralSemanticRobustnessConfig,
)
from fmeval.eval_algorithms.toxicity import Toxicity, ToxicityConfig
robustness = GeneralSemanticRobustness(
GeneralSemanticRobustnessConfig(perturbation_type="butter_finger", num_perturbations=5)
)
robustness_results = robustness.evaluate(
model=model_runner, dataset_config=data_config, num_records=100
)
toxicity = Toxicity(ToxicityConfig(model_type="detoxify"))
toxicity_results = toxicity.evaluate(
model=model_runner, dataset_config=data_config, num_records=100
)
save=True writes a per-record JSONL alongside the aggregate score. Keep it. The aggregate tells you that something regressed; the per-record file tells you which twelve prompts did, and that is the artifact that actually makes the next fine-tune better. Sort by score ascending and read the worst twenty by hand — it takes fifteen minutes and it is the highest-value quarter hour in the whole loop.
Cost note: robustness runs num_perturbations + 1 inferences per record, so 100 records with five perturbations is 600 endpoint calls. Budget accordingly, and run the expensive dimensions on a smaller slice than the accuracy pass.
Step 4: Log to MLflow so the numbers survive the notebook
A score printed to stdout is not evaluation, it is trivia. Log every run to your managed MLflow tracking server so checkpoints are comparable months apart:
import mlflow
mlflow.set_tracking_uri("arn:aws:sagemaker:us-east-1:111122223333:mlflow-tracking-server/na-mlflow")
mlflow.set_experiment("support-assistant-evals")
with mlflow.start_run(run_name="llama-3.1-8b-ft-2026-04-02"):
mlflow.log_params({
"endpoint": "llama-3-1-8b-instruct",
"base_model": "meta-llama/Llama-3.1-8B-Instruct",
"eval_dataset": "support_qa_v3",
"num_records": 300,
"temperature": 0.0,
})
for results in (qa_results, robustness_results, toxicity_results):
for r in results:
for score in r.dataset_scores:
mlflow.log_metric(f"{r.eval_name}.{score.name}", score.value)
mlflow.log_artifacts("/tmp/eval_results", artifact_path="per_record")
Now mlflow.search_runs() gives you a table of every candidate you have ever evaluated, and the MLflow UI diffs two runs side by side. When someone asks in six months why you chose this checkpoint, the answer is a URL.
Step 5: Turn the numbers into a gate
The point of automated evaluation is that a bad model cannot reach production by accident. Run the evaluation as a ProcessingStep in a SageMaker Pipeline, emit the scores as a property file, and use a ConditionStep to decide whether the RegisterModel step runs at all:
from sagemaker.workflow.properties import PropertyFile
from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo, ConditionLessThanOrEqualTo
from sagemaker.workflow.condition_step import ConditionStep
from sagemaker.workflow.functions import JsonGet
eval_report = PropertyFile(
name="EvalReport", output_name="evaluation", path="evaluation.json"
)
# step_eval = ProcessingStep(..., property_files=[eval_report])
quality_ok = ConditionGreaterThanOrEqualTo(
left=JsonGet(step_name=step_eval.name, property_file=eval_report,
json_path="qa_accuracy.f1_score"),
right=0.72,
)
safety_ok = ConditionLessThanOrEqualTo(
left=JsonGet(step_name=step_eval.name, property_file=eval_report,
json_path="toxicity.toxicity"),
right=0.01,
)
gate = ConditionStep(
name="EvalGate",
conditions=[quality_ok, safety_ok],
if_steps=[step_register],
else_steps=[],
)
Your processing script just has to write evaluation.json into the output directory in the shape those JMESPath expressions expect:
import json, pathlib
report = {
"qa_accuracy": {"f1_score": 0.781, "exact_match": 0.402},
"toxicity": {"toxicity": 0.004},
"semantic_robustness": {"word_error_rate": 0.061},
}
out = pathlib.Path("/opt/ml/processing/evaluation")
out.mkdir(parents=True, exist_ok=True)
(out / "evaluation.json").write_text(json.dumps(report))
Set the thresholds from the currently deployed model's scores, not from a round number you like. A gate at "no worse than production minus one point" is enforceable; a gate at "F1 ≥ 0.9" gets commented out the first Friday it blocks a release.
Step 6: Keep evaluating after launch
Offline evaluation predicts production quality; it does not measure it. Close the loop with three cheap habits:
- Sample live traffic. Enable data capture on the endpoint and re-run the accuracy and toxicity passes weekly against a fresh sample of real prompts. Input distributions move faster than models do.
- Shadow-test candidates. Mirror production traffic to the new checkpoint with a shadow variant and compare outputs on identical inputs before shifting a single percent of users.
- Record human verdicts. Even a thumbs-up/thumbs-down in the product, joined back to the request ID, gives you the only label that actually correlates with the business outcome. Feed the thumbs-down prompts into the next version of your eval set.
The short version
Pick 100–500 real examples with references and version them. Evaluate deterministically against the endpoint you will actually serve. Measure quality, safety, robustness, latency and cost together. Log every run to MLflow. Make the pipeline refuse to register anything that regressed against the model currently in production. That is roughly a day of work, and it converts "I think the new one is better" into a number you can defend.
Standing up evaluation and release gates around a generative workload on SageMaker AI? Our senior SageMaker consultants build these pipelines for a living — get in touch with your use case and we will help you define thresholds worth enforcing.