Multi-Region Disaster Recovery for SageMaker AI Endpoints: Warm Standby and Tested Failover

Every SageMaker AI endpoint runs in exactly one region. Multi-AZ is automatic once you run more than one instance, but a regional control-plane problem, a capacity shortfall on the GPU family you depend on, or a bad deployment that takes the whole variant down are all single-region failures. If your inference endpoint sits in a revenue path, you need a second region and a tested way to get traffic there.

This tutorial builds an active-passive (warm standby) deployment across us-east-1 and us-west-2, with cross-region model artifacts, health checks, DNS failover, and a game-day script that proves it works. The same pieces assemble into active-active if you need it.

Decide the posture first

Pick the cheapest posture that meets the recovery objective you actually have. Write the RTO and RPO down before you build anything.

Posture	RTO	Standby cost	Use when
Backup and restore	Hours	Artifacts in S3 only	Internal or batch workloads
Pilot light	15-45 min	Model + endpoint config, no endpoint	Tolerable downtime, expensive GPUs
Warm standby	1-5 min	One small instance running	Customer-facing inference
Active-active	Seconds	Full second fleet	Strict SLA, global latency needs

For GPU-backed LLM endpoints, warm standby is almost always the right answer: a standby endpoint at MinCapacity=1 on a smaller instance costs a fraction of the primary and removes the model-load delay, which is the part of recovery you cannot compress. A pilot-light GPU endpoint looks cheaper on the invoice until you measure the 8 to 15 minutes it takes to create an endpoint and pull a 16 GB model.

A note on capacity: the reason many teams discover their DR plan is broken is that the standby region does not have service quota for ml.g5.2xlarge endpoint usage. Quotas are per region and per instance type, and they default to zero for several accelerated families. Raise them in the standby region now, via Service Quotas, and verify with:

aws service-quotas list-service-quotas \
  --service-code sagemaker --region us-west-2 \
  --query "Quotas[?contains(QuotaName, 'ml.g5.2xlarge for endpoint usage')].[QuotaName,Value]" \
  --output table

A quota of 0.0 means your failover will fail at the worst possible moment.

Step 1: Replicate model artifacts and containers

SageMaker can only load a model artifact from an S3 bucket in the same region as the endpoint. Set up S3 Cross-Region Replication from your primary model bucket to a standby bucket, with replica KMS encryption:

aws s3api put-bucket-versioning --bucket my-models-use1 \
  --versioning-configuration Status=Enabled
aws s3api put-bucket-versioning --bucket my-models-usw2 \
  --versioning-configuration Status=Enabled

cat > replication.json <<'JSON'
{
  "Role": "arn:aws:iam::123456789012:role/s3-crr-models",
  "Rules": [{
    "ID": "models-to-usw2",
    "Status": "Enabled",
    "Priority": 1,
    "Filter": { "Prefix": "approved/" },
    "DeleteMarkerReplication": { "Status": "Disabled" },
    "Destination": {
      "Bucket": "arn:aws:s3:::my-models-usw2",
      "EncryptionConfiguration": {
        "ReplicaKmsKeyID": "arn:aws:kms:us-west-2:123456789012:key/REPLICA-KEY-ID"
      },
      "Metrics": { "Status": "Enabled", "EventThreshold": { "Minutes": 15 } },
      "ReplicationTime": { "Status": "Enabled", "Time": { "Minutes": 15 } }
    },
    "SourceSelectionCriteria": {
      "SseKmsEncryptedObjects": { "Status": "Enabled" }
    }
  }]
}
JSON

aws s3api put-bucket-replication --bucket my-models-use1 \
  --replication-configuration file://replication.json

Replicate only the approved/ prefix: the artifacts that passed your Model Registry gate. Replicating every training output doubles storage cost for no benefit.

Three more things cross a region boundary and are easy to forget:

• Container images. AWS Deep Learning Containers exist in every region, but the account ID and URI differ per region. Resolve them with sagemaker.image_uris.retrieve(..., region="us-west-2") rather than hardcoding. For your own BYOC images, enable ECR cross-region replication in the registry's replication configuration.
• KMS keys. A key in us-east-1 cannot decrypt in us-west-2. Either use a multi-region KMS key or give the standby region its own key and let S3 re-encrypt on replication, as above.
• Secrets. Secrets Manager supports replica secrets; create them so the standby endpoint can fetch the same Hugging Face or database credentials.

Step 2: Make the deployment regionless

Hand-built standby endpoints drift from the primary within one sprint, and drift is how failover fails. Parameterize region and deploy from the same code path. A compact Terraform module:

variable "region"        { type = string }
variable "model_data_s3" { type = string }
variable "instance_type" { type = string }
variable "min_capacity"  { type = number }

resource "aws_sagemaker_model" "llm" {
  name               = "llm-${var.region}"
  execution_role_arn = var.execution_role_arn

  primary_container {
    image          = var.container_image   # region-resolved
    model_data_url = var.model_data_s3     # region-local bucket
    environment = {
      SM_NUM_GPUS      = "1"
      MAX_TOTAL_TOKENS = "8192"
    }
  }
}

resource "aws_sagemaker_endpoint_configuration" "llm" {
  name = "llm-${var.region}-${substr(sha1(aws_sagemaker_model.llm.name), 0, 8)}"

  production_variants {
    variant_name           = "AllTraffic"
    model_name             = aws_sagemaker_model.llm.name
    initial_instance_count = var.min_capacity
    instance_type          = var.instance_type
  }
}

resource "aws_sagemaker_endpoint" "llm" {
  name                 = "llm-prod"
  endpoint_config_name = aws_sagemaker_endpoint_configuration.llm.name

  deployment_config {
    blue_green_update_policy {
      traffic_routing_configuration {
        type                     = "CANARY"
        canary_size { type = "CAPACITY_PERCENT", value = 10 }
        wait_interval_in_seconds = 300
      }
    }
  }
}

Instantiate it twice with two providers, the standby using a smaller instance_type and min_capacity = 1. Keep the endpoint name identical in both regions so client code only ever changes a region, never a name. If you already deploy endpoints from GitHub Actions, add the second region as another matrix entry; our Terraform and GitHub Actions walkthrough covers the pipeline shape.

Promotion rule to enforce in CI: a model version approved in the registry triggers deployment to primary, then to standby, in the same workflow run. Never let a model reach production in one region only.

Step 3: Put a regional front door in front of each endpoint

runtime.sagemaker.<region>.amazonaws.com is a regional endpoint and it is SigV4-signed, so you cannot simply CNAME it to a global DNS name. Give each region its own HTTPS front door that clients call, and fail over that name. The usual shape is API Gateway (regional) or an Application Load Balancer, with a small Lambda or VPC Lambda that calls invoke_endpoint:

import boto3, json, os

runtime = boto3.client("sagemaker-runtime")
ENDPOINT = os.environ["ENDPOINT_NAME"]

def handler(event, context):
    body = event.get("body") or "{}"
    resp = runtime.invoke_endpoint(
        EndpointName=ENDPOINT,
        ContentType="application/json",
        Body=body,
    )
    return {
        "statusCode": 200,
        "headers": {"content-type": "application/json",
                    "x-served-region": os.environ["AWS_REGION"]},
        "body": resp["Body"].read().decode(),
    }

The x-served-region header is not decoration; it is how you confirm during a game day that traffic actually moved.

Add a cheap health route that proves the endpoint is healthy, not just that Lambda is warm. Have it run a tiny canned inference (a two-token generation) and return 200 only if it succeeds inside a short timeout. A health check that only pings the API returns green while the model is broken.

Step 4: Route 53 health checks and failover records

HC_PRIMARY=$(aws route53 create-health-check \
  --caller-reference "infer-use1-$(date +%s)" \
  --health-check-config '{
     "Type":"HTTPS","FullyQualifiedDomainName":"use1.infer.example.com",
     "ResourcePath":"/health","Port":443,
     "RequestInterval":10,"FailureThreshold":2,
     "MeasureLatency":true
  }' --query 'HealthCheck.Id' --output text)

cat > failover.json <<JSON
{"Changes":[
 {"Action":"UPSERT","ResourceRecordSet":{
   "Name":"infer.example.com","Type":"CNAME","TTL":30,
   "SetIdentifier":"primary-use1","Failover":"PRIMARY",
   "HealthCheckId":"$HC_PRIMARY",
   "ResourceRecords":[{"Value":"use1.infer.example.com"}]}},
 {"Action":"UPSERT","ResourceRecordSet":{
   "Name":"infer.example.com","Type":"CNAME","TTL":30,
   "SetIdentifier":"standby-usw2","Failover":"SECONDARY",
   "ResourceRecords":[{"Value":"usw2.infer.example.com"}]}}
]}
JSON

aws route53 change-resource-record-sets \
  --hosted-zone-id Z123EXAMPLE --change-batch file://failover.json

A 10-second interval with a failure threshold of 2 gives roughly 30 seconds of detection, plus the 30-second TTL, plus whatever your clients cache. Java HTTP clients in particular cache DNS aggressively; set networkaddress.cache.ttl or you will fail over everywhere except in the application that matters.

If you prefer failover in code rather than DNS, the alternative is a client-side wrapper that retries the standby region on 5xx or timeout. It is faster and it is also one more thing to maintain in every language your clients use. DNS failover plus a short retry budget covers most teams.

Step 5: Scale the standby on promotion

A standby at one small instance will collapse under full production traffic. Failover must include a capacity step. Trigger it from the Route 53 health check alarm:

import boto3

asg = boto3.client("application-autoscaling", region_name="us-west-2")
sm  = boto3.client("sagemaker", region_name="us-west-2")

def promote(event, context):
    # 1. widen the autoscaling bounds
    asg.register_scalable_target(
        ServiceNamespace="sagemaker",
        ResourceId="endpoint/llm-prod/variant/AllTraffic",
        ScalableDimension="sagemaker:variant:DesiredInstanceCount",
        MinCapacity=4, MaxCapacity=16,
    )
    # 2. jump straight to a floor that can absorb the shift
    sm.update_endpoint_weights_and_capacities(
        EndpointName="llm-prod",
        DesiredWeightsAndCapacities=[
            {"VariantName": "AllTraffic", "DesiredInstanceCount": 4}
        ],
    )

Target tracking alone reacts too slowly here: scale-out on a GPU instance includes a multi-minute model load, and by the time the metric moves you have already shed requests. Set the floor explicitly, then let autoscaling take over.

Budget for the standby being expensive once promoted, and add a documented step to scale it back down after the incident. More than one team has discovered their DR event cost them three weeks of GPU spend because nobody turned the floor back off.

Step 6: Run a game day, on a schedule

An untested failover is a hypothesis. Run this quarterly, in business hours, with the team watching:

1. Record the baseline: p95 latency, error rate, x-served-region on live traffic.
2. Break the primary deliberately. The cleanest lever is failing the health check: have /health return 503 behind a feature flag, or associate a deny-all security group with the primary front door. Do not delete the primary endpoint.
3. Start a stopwatch. Watch for the Route 53 health check to go unhealthy, the record to flip, and x-served-region to change on real traffic.
4. Confirm the promotion Lambda widened capacity and that latency returns to baseline.
5. Compare outputs: send the same 50 prompts to both regions and diff. Divergence means artifact drift, usually a standby that is one model version behind.
6. Fail back deliberately, scale the standby down, and write up measured RTO against the objective.

Things game days routinely surface: a missing quota in the standby region, an IAM execution role that exists only in the primary account path, a KMS key the standby role cannot use, a VPC endpoint missing so the standby Lambda cannot reach runtime.sagemaker, and a model artifact that replicated before it was approved.

What this does not cover

Multi-region DR protects availability, not correctness. You still need drift detection in both regions (data capture and Model Monitor run per region, so deploy both), and the standby's capture bucket needs the same retention policy for audit purposes. Feature stores and vector stores need their own replication story; an endpoint that fails over to a region whose OpenSearch collection is empty is a working endpoint serving wrong answers.

And if the workload is asynchronous or batch, most of this is unnecessary: queue the work, let it drain when the region recovers, and spend the money on something else.

Summary

The working parts of a credible SageMaker AI DR plan are a written RTO, quota confirmed in the standby region, replicated artifacts and images, one deployment code path for both regions, a regional front door with a real inference health check, DNS failover with short TTLs, an explicit capacity promotion step, and a quarterly game day.

If you want a second set of eyes on a resilience design, or help running the first game day, get in touch.