NeuralArmada | Zero-Downtime Endpoint Updates: Blue/Green Guardrails, Canary Shifting, and Shadow Tests

Every SageMaker AI team eventually ships a model update that looks fine in evaluation and is wrong in production: a tokenizer version bump that changes truncation, a container upgrade that halves throughput, a feature that silently arrives as null. Offline metrics do not catch these. Deployment guardrails and shadow tests do.

This tutorial covers the three mechanisms SageMaker AI gives you for changing a live endpoint without a maintenance window — blue/green with canary traffic shifting, rolling updates, and shadow (mirror) testing — with working boto3 code, the CloudWatch alarms that make auto-rollback real, and the failure modes we see most often on client engagements.

Assumptions: a real-time endpoint already exists, you have boto3 and permission to call sagemaker:UpdateEndpoint, and there is a second model package or model artifact you want to roll out.

The three mechanisms, and when to use each

Mechanism	What it does	Use it when
Blue/green + canary	Spins up a full new fleet, shifts traffic in steps, keeps the old fleet warm for a rollback window	Default for any endpoint that serves customer traffic
Rolling update	Replaces capacity in batches on the existing endpoint	Large fleets where doubling capacity is impossible or too expensive
Shadow test	Mirrors a copy of live traffic to a candidate variant whose responses are discarded	You need production-traffic evidence before any user sees the new model

The order we recommend on real projects: shadow test first for a day or two, then blue/green canary the winner. Shadow answers "does it behave", canary answers "does it survive".

Step 1: Alarms first, deployment second

Auto-rollback is only as good as the alarms attached to it. Create them before the update, on the endpoint you are about to change. Two alarms cover most cases: model errors and latency.

import boto3

cw = boto3.client("cloudwatch")
ENDPOINT = "fraud-scoring-prod"
VARIANT = "AllTraffic"

cw.put_metric_alarm(
    AlarmName=f"{ENDPOINT}-5xx",
    Namespace="AWS/SageMaker",
    MetricName="Invocation5XXErrors",
    Dimensions=[
        {"Name": "EndpointName", "Value": ENDPOINT},
        {"Name": "VariantName", "Value": VARIANT},
    ],
    Statistic="Sum",
    Period=60,
    EvaluationPeriods=1,
    Threshold=5,
    ComparisonOperator="GreaterThanOrEqualToThreshold",
    TreatMissingData="notBreaching",
)

cw.put_metric_alarm(
    AlarmName=f"{ENDPOINT}-p99-latency",
    Namespace="AWS/SageMaker",
    MetricName="ModelLatency",
    Dimensions=[
        {"Name": "EndpointName", "Value": ENDPOINT},
        {"Name": "VariantName", "Value": VARIANT},
    ],
    ExtendedStatistic="p99",
    Period=60,
    EvaluationPeriods=2,
    Threshold=800_000,  # microseconds — ModelLatency is in microseconds
    ComparisonOperator="GreaterThanThreshold",
    TreatMissingData="notBreaching",
)

Two details that bite people:

ModelLatency is reported in microseconds, not milliseconds. A threshold of 800 means 0.8 ms and will fire constantly.
TreatMissingData="notBreaching" matters. During a canary step the new fleet may publish no datapoints for a minute; the default (missing) can be interpreted as a breach and roll you back for no reason.

Alarms used for rollback must be in OK or INSUFFICIENT_DATA state when the update starts. An alarm already in ALARM state will abort the deployment immediately.

Step 2: A new endpoint config for the candidate

Deployment guardrails are applied on update_endpoint, so you need a second endpoint config pointing at the new model.

sm = boto3.client("sagemaker")

sm.create_model(
    ModelName="fraud-scoring-v7",
    ExecutionRoleArn=ROLE_ARN,
    PrimaryContainer={
        "Image": IMAGE_URI,
        "ModelDataUrl": "s3://my-models/fraud/v7/model.tar.gz",
        "Environment": {"MODEL_VERSION": "7"},
    },
)

sm.create_endpoint_config(
    EndpointConfigName="fraud-scoring-v7-cfg",
    ProductionVariants=[
        {
            "VariantName": "AllTraffic",
            "ModelName": "fraud-scoring-v7",
            "InstanceType": "ml.c7i.2xlarge",
            "InitialInstanceCount": 4,
            "RoutingConfig": {"RoutingStrategy": "LEAST_OUTSTANDING_REQUESTS"},
        }
    ],
)

Keep the variant name identical to the current one. If you rename the variant, your CloudWatch alarms (which are scoped by VariantName) stop matching, autoscaling targets break, and dashboards go blank right when you need them.

Step 3: Blue/green with canary traffic shifting

sm.update_endpoint(
    EndpointName=ENDPOINT,
    EndpointConfigName="fraud-scoring-v7-cfg",
    RetainAllVariantProperties=True,
    DeploymentConfig={
        "BlueGreenUpdatePolicy": {
            "TrafficRoutingConfiguration": {
                "Type": "CANARY",
                "CanarySize": {"Type": "CAPACITY_PERCENT", "Value": 10},
                "WaitIntervalInSeconds": 600,
            },
            "TerminationWaitInSeconds": 900,
            "MaximumExecutionTimeoutInSeconds": 3600,
        },
        "AutoRollbackConfiguration": {
            "Alarms": [
                {"AlarmName": f"{ENDPOINT}-5xx"},
                {"AlarmName": f"{ENDPOINT}-p99-latency"},
            ]
        },
    },
)

What actually happens:

SageMaker provisions the green fleet at canary size (10 percent of capacity here).
It shifts 10 percent of traffic to green and bakes for WaitIntervalInSeconds while watching your alarms.
If nothing fires, it provisions the rest of green and shifts 100 percent.
Blue stays alive for TerminationWaitInSeconds — this is your manual rollback window — then is deleted.

RetainAllVariantProperties=True preserves current instance counts set by autoscaling instead of resetting to InitialInstanceCount. Forget it and a scaled-out endpoint drops back to four instances at the worst possible moment.

For a gentler ramp, use LINEAR instead:

"TrafficRoutingConfiguration": {
    "Type": "LINEAR",
    "LinearStepSize": {"Type": "CAPACITY_PERCENT", "Value": 20},
    "WaitIntervalInSeconds": 300,
},

Five steps of 20 percent with a five-minute bake each: roughly 25 minutes of exposure, and any step can trigger rollback. ALL_AT_ONCE still gets you the blue/green safety net (old fleet retained, alarms watched) without a partial-traffic phase — acceptable for internal endpoints, not for revenue paths.

Budget capacity before you start: blue/green temporarily needs room for both fleets. If your account is at its ml.c7i.2xlarge endpoint-usage quota, the update fails at provisioning. Check quotas, or use a rolling update.

Step 4: Rolling updates when you cannot double capacity

sm.update_endpoint(
    EndpointName=ENDPOINT,
    EndpointConfigName="fraud-scoring-v7-cfg",
    RetainAllVariantProperties=True,
    DeploymentConfig={
        "RollingUpdatePolicy": {
            "MaximumBatchSize": {"Type": "CAPACITY_PERCENT", "Value": 20},
            "WaitIntervalInSeconds": 300,
            "MaximumExecutionTimeoutInSeconds": 3600,
            "RollbackMaximumBatchSize": {"Type": "CAPACITY_PERCENT", "Value": 50},
        },
        "AutoRollbackConfiguration": {"Alarms": [{"AlarmName": f"{ENDPOINT}-5xx"}]},
    },
)

This replaces instances in batches, so peak extra capacity is one batch rather than a whole fleet. The trade-off is that during the rollout, both model versions serve real traffic simultaneously and there is no single warm fleet to snap back to — rollback is itself a batched operation. Use it for large fleets or quota-constrained accounts, not as the default.

Step 5: Shadow tests — evidence before exposure

A shadow test sends a copy of live requests to a candidate variant and throws the responses away. Users never see them; you get production-distribution latency and error data, plus captured inputs and outputs you can diff offline.

First, add the candidate as a shadow variant in the endpoint config:

sm.create_endpoint_config(
    EndpointConfigName="fraud-scoring-shadow-cfg",
    ProductionVariants=[
        {
            "VariantName": "AllTraffic",
            "ModelName": "fraud-scoring-v6",
            "InstanceType": "ml.c7i.2xlarge",
            "InitialInstanceCount": 4,
        }
    ],
    ShadowProductionVariants=[
        {
            "VariantName": "shadow-v7",
            "ModelName": "fraud-scoring-v7",
            "InstanceType": "ml.c7i.2xlarge",
            "InitialInstanceCount": 1,
        }
    ],
)

Then run it as a managed inference experiment, which handles the schedule, the sampling percentage, and data capture:

sm.create_inference_experiment(
    Name="fraud-v7-shadow",
    Type="ShadowMode",
    RoleArn=ROLE_ARN,
    EndpointName=ENDPOINT,
    ModelVariants=[
        {
            "ModelName": "fraud-scoring-v6",
            "VariantName": "AllTraffic",
            "InfrastructureConfig": {
                "InfrastructureType": "RealTimeInference",
                "RealTimeInferenceConfig": {
                    "InstanceType": "ml.c7i.2xlarge",
                    "InstanceCount": 4,
                },
            },
        },
        {
            "ModelName": "fraud-scoring-v7",
            "VariantName": "shadow-v7",
            "InfrastructureConfig": {
                "InfrastructureType": "RealTimeInference",
                "RealTimeInferenceConfig": {
                    "InstanceType": "ml.c7i.2xlarge",
                    "InstanceCount": 1,
                },
            },
        },
    ],
    ShadowModeConfig={
        "SourceModelVariantName": "AllTraffic",
        "ShadowModelVariants": [
            {"ShadowModelVariantName": "shadow-v7", "SamplingPercentage": 20}
        ],
    },
    DataStorageConfig={"Destination": "s3://my-ml-logs/shadow/fraud-v7/"},
)

Start at 10–20 percent sampling, not 100. You are paying for the shadow instances, and 20 percent of production traffic is plenty to expose a latency regression or a systematic prediction shift.

What to compare once data lands in S3:

Latency: p50/p90/p99 of ModelLatency per variant in CloudWatch, not averages.
Errors: any Invocation5XXErrors on the shadow variant is a hard stop.
Prediction agreement: join captured input/output records by request ID and measure disagreement rate. A 30 percent disagreement on a fraud score is not "the new model is better", it is a bug until proven otherwise.
Input assumptions: check that fields the new model expects are actually present in live payloads. This is where feature-engineering drift shows up.

Stop the experiment explicitly when you are done — shadow instances keep billing until you do:

sm.stop_inference_experiment(
    Name="fraud-v7-shadow",
    ModelVariantActions={"shadow-v7": "Remove"},
)

Troubleshooting

Symptom	Likely cause	Fix
Update fails instantly with a validation error about alarms	A rollback alarm is already in `ALARM` state, or is in another region	Resolve the alarm first; alarms must be in the endpoint's region
Rollback triggers during every canary step	`TreatMissingData` default plus a sparse metric	Set `TreatMissingData="notBreaching"` and require 2 evaluation periods
`ResourceLimitExceeded` at green provisioning	Endpoint-usage quota cannot fit both fleets	Request a quota increase or switch to `RollingUpdatePolicy`
Endpoint drops to `InitialInstanceCount` after update	`RetainAllVariantProperties` omitted	Always pass `RetainAllVariantProperties=True` on autoscaled endpoints
Deployment times out mid-rollout	Large artifact plus slow container start exceeds `MaximumExecutionTimeoutInSeconds`	Raise the timeout and the container health-check grace period
Alarms never fire but users report errors	Errors are returned as HTTP 200 with an error body	Emit a custom CloudWatch metric from the inference handler and alarm on that

Checklist

Alarms created and healthy before update_endpoint, with microsecond-correct latency thresholds.
Variant name unchanged between old and new endpoint configs.
CANARY or LINEAR routing with a bake interval long enough to produce datapoints (5–10 minutes minimum).
TerminationWaitInSeconds set to a real human rollback window, not 0.
RetainAllVariantProperties=True on any autoscaled endpoint.
Capacity quota checked for double-fleet headroom, or rolling update chosen instead.
Shadow test at 10–20 percent sampling with data capture, compared on latency, errors, and prediction agreement.
Explicit teardown of shadow variants and inference experiments.

Where this fits

Guardrails are the missing half of an MLOps loop: the registry decides what gets approved, deployment guardrails decide how it reaches traffic, and Model Monitor watches what happens afterwards. Teams that add all three stop treating model releases as events that need a weekend.

Want a review of your deployment path before the next model version ships — or help wiring canary and shadow stages into an existing pipeline? See our MLOps consulting page, or talk to us.