+1 (726) 227-3241
NeuralArmada

Ship SageMaker AI Endpoints from Git: Terraform and GitHub Actions

Most SageMaker AI teams reach the same wall at roughly the same time. The model works, the pipeline runs, and the endpoint exists — but nobody can say exactly how that endpoint got its current configuration, which IAM role it assumes, or what would happen if it were deleted at 2 a.m. The notebook that created it has a variable called role and a comment that says # TODO.

This tutorial closes that gap. We put the serving side of SageMaker AI — model, endpoint config, endpoint, autoscaling, alarms — into Terraform, and we drive the deploys from GitHub Actions using short-lived OIDC credentials instead of stored AWS keys. The model artifact itself still comes from the Model Registry, so this composes with the training pipeline loop rather than replacing it.

The split matters, so state it up front:

  • Pipelines / Model Registry own the artifact. What was trained, on which data, with which metrics, and whether a human approved it.
  • Terraform owns the infrastructure. Which approved package is currently serving, on what instance type, behind which scaling policy and alarms.

Mixing those two is the usual failure mode. Terraform should never train a model, and a pipeline should never hand-roll an endpoint in a way Terraform cannot see.

Prerequisites

  • Terraform 1.9+ and an S3 backend with state locking (S3 native locking with use_lockfile = true works now; a DynamoDB table is still fine).
  • A Model Package Group in the SageMaker Model Registry with at least one Approved package.
  • Permission to create an IAM OIDC identity provider in the target account.

Step 1: GitHub OIDC instead of access keys

Long-lived AWS_ACCESS_KEY_ID secrets in a CI system are the single most common finding in the security reviews we run. Replace them with a federated role. Create the provider and role once, in a bootstrap Terraform stack:

data "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"
}

data "aws_iam_policy_document" "assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [data.aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    # Scope to one repo AND one ref. Without the sub condition any
    # repository on GitHub can assume this role.
    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:acme/ml-serving:ref:refs/heads/main"]
    }
  }
}

resource "aws_iam_role" "deployer" {
  name               = "sagemaker-endpoint-deployer"
  assume_role_policy = data.aws_iam_policy_document.assume.json
}

If you allow pull-request workflows to assume a role, give them a separate, read-only role scoped to repo:acme/ml-serving:pull_request. A plan job does not need sagemaker:CreateEndpoint.

Step 2: Resolve the latest approved package

Terraform should not hardcode a model package ARN — that turns every model promotion into a hand edit. Look it up instead. The cleanest way is a small external data source or a variable fed by the workflow; here is the variable approach, which keeps the plan diff honest because the ARN shows up in the plan output:

aws sagemaker list-model-packages \
  --model-package-group-name churn-xgboost \
  --model-approval-status Approved \
  --sort-by CreationTime --sort-order Descending \
  --max-results 1 \
  --query 'ModelPackageSummaryList[0].ModelPackageArn' --output text

The workflow captures that string and passes it as TF_VAR_model_package_arn. Now "promote a model" is a Terraform plan that changes exactly one input, and the review shows which package is replacing which.

Step 3: The endpoint stack

variable "model_package_arn" { type = string }
variable "endpoint_name"     { default = "churn-prod" }

resource "aws_sagemaker_model" "this" {
  name                 = "churn-${substr(sha1(var.model_package_arn), 0, 12)}"
  execution_role_arn   = aws_iam_role.execution.arn

  primary_container {
    model_package_name = var.model_package_arn
  }

  vpc_config {
    subnets            = var.private_subnet_ids
    security_group_ids = [aws_security_group.endpoint.id]
  }

  tags = local.tags
}

resource "aws_sagemaker_endpoint_configuration" "this" {
  name       = "${var.endpoint_name}-${substr(sha1(var.model_package_arn), 0, 12)}"
  kms_key_arn = aws_kms_key.endpoint.arn

  production_variants {
    variant_name           = "AllTraffic"
    model_name             = aws_sagemaker_model.this.name
    initial_instance_count = 2
    instance_type          = "ml.m5.large"
  }

  data_capture_config {
    enable_capture              = true
    initial_sampling_percentage = 20
    destination_s3_uri          = "s3://${var.capture_bucket}/churn-prod"

    capture_options { capture_mode = "Input" }
    capture_options { capture_mode = "Output" }
  }

  lifecycle { create_before_destroy = true }
}

Three details do most of the work here:

  1. Names derived from the artifact hash. SageMaker model and endpoint-config resources are immutable; if the name is static, Terraform must destroy and recreate, which means downtime. Hashing the package ARN into the name makes each promotion a new resource.
  2. create_before_destroy. Combined with the derived name, this gives you the new config before the old one goes away.
  3. Data capture on from day one. Turning it on later means a config replacement anyway, and Model Monitor has no history to work with.

Now the endpoint itself, with managed blue/green rollout:

resource "aws_sagemaker_endpoint" "this" {
  name                 = var.endpoint_name
  endpoint_config_name = aws_sagemaker_endpoint_configuration.this.name

  deployment_config {
    blue_green_update_policy {
      traffic_routing_configuration {
        type                     = "CANARY"
        wait_interval_in_seconds = 600

        canary_size {
          type  = "CAPACITY_PERCENT"
          value = 20
        }
      }
      termination_wait_in_seconds = 300
    }

    auto_rollback_configuration {
      alarms { alarm_name = aws_cloudwatch_metric_alarm.errors.alarm_name }
      alarms { alarm_name = aws_cloudwatch_metric_alarm.latency.alarm_name }
    }
  }

  tags = local.tags
}

An auto_rollback_configuration with no alarms attached is decoration. The alarms must be in ALARM state during the canary window for the rollback to fire, so keep the evaluation period short (one minute, one or two datapoints) and the threshold tight; the mechanics are covered in detail in our post on zero-downtime endpoint updates.

Scaling belongs in the same stack:

resource "aws_appautoscaling_target" "variant" {
  service_namespace  = "sagemaker"
  resource_id        = "endpoint/${aws_sagemaker_endpoint.this.name}/variant/AllTraffic"
  scalable_dimension = "sagemaker:variant:DesiredInstanceCount"
  min_capacity       = 2
  max_capacity       = 8
}

resource "aws_appautoscaling_policy" "invocations" {
  name               = "churn-invocations"
  policy_type        = "TargetTrackingScaling"
  service_namespace  = aws_appautoscaling_target.variant.service_namespace
  resource_id        = aws_appautoscaling_target.variant.resource_id
  scalable_dimension = aws_appautoscaling_target.variant.scalable_dimension

  target_tracking_scaling_policy_configuration {
    target_value       = 70
    scale_in_cooldown  = 300
    scale_out_cooldown = 60

    predefined_metric_specification {
      predefined_metric_type = "SageMakerVariantInvocationsPerInstance"
    }
  }
}

Step 4: The workflow

name: deploy-endpoint
on:
  push:
    branches: [main]
    paths: ["infra/**", ".github/workflows/deploy-endpoint.yml"]
  workflow_dispatch:

permissions:
  id-token: write   # required for OIDC
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production   # gate with required reviewers
    defaults:
      run: { working-directory: infra }
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/sagemaker-endpoint-deployer
          aws-region: us-east-1

      - name: Resolve approved model package
        run: |
          ARN=$(aws sagemaker list-model-packages \
            --model-package-group-name churn-xgboost \
            --model-approval-status Approved \
            --sort-by CreationTime --sort-order Descending --max-results 1 \
            --query 'ModelPackageSummaryList[0].ModelPackageArn' --output text)
          test "$ARN" != "None"
          echo "TF_VAR_model_package_arn=$ARN" >> "$GITHUB_ENV"

      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
      - run: terraform plan -out=tfplan
      - run: terraform apply -auto-approve tfplan

      - name: Smoke test
        run: |
          aws sagemaker-runtime invoke-endpoint \
            --endpoint-name churn-prod \
            --content-type text/csv --body fileb://../tests/sample.csv \
            /tmp/out.json
          cat /tmp/out.json

Use a GitHub environment with required reviewers for the production job. That gives you the human approval gate without a human touching the console — the reviewer approves a plan, and the plan is the change.

Step 5: Drift, and what not to put in Terraform

Terraform's value collapses the first time someone updates the endpoint by hand. Add a scheduled workflow that runs terraform plan -detailed-exitcode nightly and fails on exit code 2, then alerts. Two days of unnoticed drift is recoverable; two months is a rewrite.

Some things genuinely do not belong in this stack:

  • Training jobs and pipeline executions. They are runs, not resources. Terraform can create the pipeline definition; it should not start it.
  • Model artifacts. They live in S3 and the registry, versioned there.
  • Anything the autoscaler owns. Do not manage initial_instance_count drift with terraform apply; set it once and let target tracking move it. Add ignore_changes if your provider version surfaces it.
  • Secrets. Use Secrets Manager or SSM Parameter Store and reference by ARN.

A workable repository layout

infra/
  backend.tf          # S3 state, per-environment key
  endpoint.tf         # model, config, endpoint, autoscaling
  alarms.tf           # rollback + paging alarms
  iam.tf              # execution role, least privilege
  envs/
    staging.tfvars
    prod.tfvars
tests/
  sample.csv
.github/workflows/
  plan.yml            # PRs: read-only role, plan only
  deploy-endpoint.yml # main: apply behind environment approval

Run staging and prod as separate state files, not separate workspaces sharing one backend key. When a promotion goes wrong at 2 a.m., you want the blast radius to be one file.

Where this usually takes teams

For a single account and one endpoint this is about a two-day build. Where it stretches is the IAM work: the execution role, the deployer role, and the KMS grants take longer than all the SageMaker resources combined, especially in a VPC-only environment. Budget for that, and reuse the module for every endpoint after the first — the second one should be a .tfvars file.

We build this deployment path as part of most MLOps consulting engagements, and it is usually the fastest way to make an ML platform auditable. If you want a second pair of eyes on your own Terraform or CI setup, get in touch.