Detect Drift in Production: Data Capture, Model Monitor, and CloudWatch Alarms

Most SageMaker AI endpoints ship without monitoring, and the failure mode is quiet: the model keeps returning 200s while its inputs drift away from the data it was trained on and accuracy erodes for months before anyone notices. This tutorial closes that loop. You turn on data capture, generate a baseline from your training set, schedule a data quality monitor and a model quality monitor, and wire the violation counts to a CloudWatch alarm that pages a human.

Everything below uses the SageMaker Python SDK v2 (pip install "sagemaker>=2.200") and assumes you already have a real-time endpoint. If you do not, start with deploying a Hugging Face model to a real-time endpoint.

The four monitor types

SageMaker Model Monitor runs scheduled processing jobs that compare recent production traffic against a baseline and emit a violations report.

Monitor	Detects	Needs ground truth?
Data quality	Schema changes, missing values, distribution shift in features	No
Model quality	Accuracy, RMSE, F1 drift against actual outcomes	Yes
Bias drift (Clarify)	Change in fairness metrics across groups	Labels for the metric you chose
Feature attribution drift (Clarify)	Change in SHAP feature importance ranking	No

Start with data quality. It needs nothing but traffic, and it catches the majority of real incidents: an upstream ETL job that starts sending nulls, a client that switches units, a categorical feature that gains a new level.

1. Turn on data capture

Capture is a property of the endpoint configuration, so enabling it on an existing endpoint means an update, not a redeploy of the model.

from sagemaker.model_monitor import DataCaptureConfig
from sagemaker.predictor import Predictor

bucket = "my-ml-artifacts"
capture_uri = f"s3://{bucket}/monitoring/datacapture"

predictor = Predictor(endpoint_name="fraud-scorer")
predictor.update_data_capture_config(
    data_capture_config=DataCaptureConfig(
        enable_capture=True,
        sampling_percentage=100,
        destination_s3_uri=capture_uri,
        capture_options=["REQUEST", "RESPONSE"],
    )
)

Notes that save time later:

• sampling_percentage=100 is fine while you validate the setup. On a high-volume endpoint drop it to 10 or 20 once monitors are green; the capture files themselves cost money and the statistics do not need every record.
• Captured objects land under .../<endpoint>/<variant>/<yyyy>/<mm>/<dd>/<hh>/ as JSON Lines. The hourly partitioning is what monitoring schedules read, so nothing appears to a monitor until an hour has closed.
• Add an S3 lifecycle rule that expires capture data after 30 to 90 days. Forgotten capture buckets are one of the most common surprise line items on an ML account.
• The endpoint execution role needs s3:PutObject on the capture prefix.

2. Baseline from your training data

The baseline job runs Deequ over your training set and produces two artifacts: statistics.json (per-feature distributions) and constraints.json (the rules that production data will be checked against).

from sagemaker.model_monitor import DefaultModelMonitor
from sagemaker.model_monitor.dataset_format import DatasetFormat
import sagemaker

role = sagemaker.get_execution_role()
baseline_uri = f"s3://{bucket}/monitoring/baseline"

monitor = DefaultModelMonitor(
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    volume_size_in_gb=20,
    max_runtime_in_seconds=1800,
)

monitor.suggest_baseline(
    baseline_dataset=f"s3://{bucket}/train/train.csv",
    dataset_format=DatasetFormat.csv(header=True),
    output_s3_uri=baseline_uri,
    wait=True,
)

The baseline dataset must have the same column order the endpoint receives, with the target column present and named. Open the suggested constraints before you trust them:

constraints = monitor.suggested_constraints()
for feature in constraints.body_dict["features"][:5]:
    print(feature["name"], feature["inferred_type"], feature["completeness"])

Suggested constraints are a starting point, not a policy. Two edits are almost always worth making by hand:

1. Relax completeness on features that are legitimately sparse. A feature that is 60 percent null in training will be flagged every hour otherwise.
2. Set monitoring_config.datatype_check_threshold and domain_content.threshold deliberately. The defaults are strict enough that a single new categorical value trips a violation, which is right for a payments model and noise for a recommendation model.

Edit the JSON, write it back to S3, and pass that URI as constraints in the next step.

3. Schedule the data quality monitor

from sagemaker.model_monitor import CronExpressionGenerator

monitor.create_monitoring_schedule(
    monitor_schedule_name="fraud-scorer-data-quality",
    endpoint_input=predictor.endpoint_name,
    output_s3_uri=f"s3://{bucket}/monitoring/reports",
    statistics=monitor.baseline_statistics(),
    constraints=monitor.suggested_constraints(),
    schedule_cron_expression=CronExpressionGenerator.hourly(),
    enable_cloudwatch_metrics=True,
)

Hourly is the shortest supported interval and the right default for a production endpoint. Daily is enough for a batch-scored model and costs a twenty-fourth as much in processing time. Each execution is a billed processing job, so an ml.m5.xlarge running hourly is roughly the cost of a small always-on instance: budget for it, and use daily schedules on low-traffic endpoints.

Check the first execution:

import time

desc = monitor.describe_schedule()
print(desc["MonitoringScheduleStatus"])

executions = monitor.list_executions()
latest = executions[-1]
print(latest.describe()["ProcessingJobStatus"])

violations = latest.constraint_violations()
if violations:
    for v in violations.body_dict["violations"]:
        print(v["feature_name"], v["constraint_check_type"], v["description"])

A schedule with no traffic in the hour produces a Failed execution with the message "Job inputs had no data". That is expected on a quiet endpoint, not a broken setup, but it does mean an alarm on execution failure alone will be noisy.

4. Add model quality when you have labels

Data quality tells you the inputs changed. Model quality tells you whether the predictions got worse, and it needs ground truth: the actual outcome of each prediction, uploaded to S3 and joined by an inference ID.

Pass the ID on every request:

response = runtime.invoke_endpoint(
    EndpointName="fraud-scorer",
    ContentType="text/csv",
    Body=payload,
    InferenceId="txn-90210",
)

Then write ground truth records into an hourly prefix as they become known:

{"groundTruthData": {"data": "1", "encoding": "CSV"},
 "eventMetadata": {"eventId": "txn-90210"},
 "eventVersion": "0"}

from sagemaker.model_monitor import ModelQualityMonitor, EndpointInput

mq = ModelQualityMonitor(
    role=role, instance_count=1, instance_type="ml.m5.xlarge",
    max_runtime_in_seconds=1800,
)

mq.suggest_baseline(
    baseline_dataset=f"s3://{bucket}/validation/predictions.csv",
    dataset_format=DatasetFormat.csv(header=True),
    problem_type="BinaryClassification",
    inference_attribute="prediction",
    probability_attribute="probability",
    ground_truth_attribute="label",
    output_s3_uri=f"s3://{bucket}/monitoring/mq-baseline",
)

mq.create_monitoring_schedule(
    monitor_schedule_name="fraud-scorer-model-quality",
    endpoint_input=EndpointInput(
        endpoint_name="fraud-scorer",
        destination="/opt/ml/processing/input_data",
        probability_attribute="0",
        probability_threshold_attribute=0.5,
    ),
    ground_truth_input=f"s3://{bucket}/monitoring/groundtruth",
    problem_type="BinaryClassification",
    output_s3_uri=f"s3://{bucket}/monitoring/reports",
    constraints=mq.suggested_constraints(),
    schedule_cron_expression=CronExpressionGenerator.daily(),
    enable_cloudwatch_metrics=True,
)

The practical constraint is label latency. If a fraud label arrives 45 days after the transaction, a daily model quality monitor is reporting on decisions you made six weeks ago. That is still worth having, but the alert that protects you day to day is the data quality one.

5. Alarm on it, or it did not happen

With enable_cloudwatch_metrics=True, monitors publish to the aws/sagemaker/Endpoints/data-metrics namespace. The metric that matters is the violation count:

import boto3

cw = boto3.client("cloudwatch")
cw.put_metric_alarm(
    AlarmName="fraud-scorer-drift",
    Namespace="aws/sagemaker/Endpoints/data-metrics",
    MetricName="feature_baseline_drift_amount_spend_30d",
    Dimensions=[
        {"Name": "Endpoint", "Value": "fraud-scorer"},
        {"Name": "MonitoringSchedule", "Value": "fraud-scorer-data-quality"},
    ],
    Statistic="Maximum",
    Period=3600,
    EvaluationPeriods=2,
    Threshold=0.4,
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789012:ml-oncall"],
    TreatMissingData="notBreaching",
)

Per-feature drift metrics are named feature_baseline_drift_<feature_name>, so a wide model produces a lot of them. Two patterns work better than alarming on every feature:

• Alarm on the handful of features the model actually depends on, taken from the SHAP ranking in your Clarify explainability report.
• Route the EventBridge event SageMaker Model Monitor Execution Status Change with "CompletedWithViolations" to a Lambda that summarizes the violations report into one Slack message per execution.

Requiring two consecutive breaching periods (EvaluationPeriods=2) matters more than the exact threshold. Single-hour drift spikes are usually a batch of odd traffic, not a model problem.

6. What to do when it fires

A drift alert is a triage prompt, not an automatic retrain trigger. Work through it in this order:

1. Is it a data bug? Check the upstream pipeline first. A new null, a unit change, a renamed category, or a client sending unscaled values explains most alerts and is not a model problem at all.
2. Is it seasonality? Compare against the same window last month. Retraining on a seasonal shift bakes the season into the model.
3. Is it real population change? Then retrain, and treat the current traffic window as part of your new training set.

When the answer is retrain, the fix should run through a pipeline with a registry approval rather than a notebook. Wire the EventBridge event to pipeline.start() and let the evaluation step decide whether the new model is actually better; see A Minimal MLOps Loop for that pipeline.

Monitoring LLM and generative endpoints

Classic Model Monitor assumes tabular features, so it does not usefully baseline free text. For an LLM endpoint hosted on SageMaker AI, the equivalent controls are different:

• Capture requests and responses anyway; they are the raw material for evaluation sets and incident review.
• Score a sample of traffic offline against an evaluation harness (relevance, faithfulness, toxicity) on a schedule, and alarm on the score, not on feature statistics.
• Track embedding drift of the input prompts as a proxy for "our users are asking different things now".
• Watch operational metrics that correlate with quality: ModelLatency, truncation rates, and the share of requests hitting your max token limit.

If your generative workload sits on Bedrock rather than SageMaker AI, the monitoring surface is different again; we compare the two in Bedrock or SageMaker AI for Generative AI.

Cost and cleanup

Monitoring is not free: each schedule execution is a processing job, plus S3 for capture and reports. A realistic setup is one hourly data quality schedule and one daily model quality schedule per production endpoint. Keep development endpoints out of it entirely.

monitor.delete_monitoring_schedule()
mq.delete_monitoring_schedule()

Deleting the endpoint does not delete its schedules, and orphaned schedules keep launching failing jobs. Add the two lines above to whatever teardown script deletes the endpoint.

Checklist

• Data capture on, sampled sensibly, with an S3 lifecycle rule.
• A baseline built from training data, with suggested constraints reviewed by a human.
• An hourly data quality schedule and, if labels exist, a daily model quality schedule.
• A CloudWatch alarm on the two or three features that matter, requiring two consecutive periods.
• An EventBridge rule that turns CompletedWithViolations into a message a person reads.
• Teardown that removes schedules along with the endpoint.

Want a second pair of eyes on drift monitoring for a model already in production? Talk to us.