NeuralArmada | Private by Default: VPC-Only SageMaker AI with PrivateLink, KMS, and IAM Guardrails

Sooner or later every SageMaker engagement meets the security review. Someone asks where the training data goes when a job runs, whether a notebook can curl an arbitrary host, and which key encrypted the model artifact. If the answer is "AWS handles it," the project stalls.

This tutorial builds the answer: a SageMaker AI setup where Studio has no route to the internet, training and inference traffic never leaves your VPC, every artifact is encrypted with a customer-managed KMS key, and IAM refuses to create resources that break those rules. It is the configuration we deploy for clients in regulated industries, and it works just as well for a startup that simply does not want customer data on a public path.

The four layers

Private SageMaker is not one setting. It is four that have to agree:

Network placement — domains, jobs, and endpoints attached to private subnets.
VPC endpoints — because SageMaker's own control plane, ECR, S3, CloudWatch, and STS are all reached over the network, and with no NAT gateway they must be reachable privately.
Encryption — customer-managed KMS keys on volumes, outputs, and buckets.
IAM guardrails — conditions that make the safe configuration the only configuration.

Skip any one of them and you get either a job that hangs in Starting for 20 minutes and then fails, or a compliant-looking setup with a quiet hole in it.

Step 1: VPC and subnets

Use at least two private subnets in different Availability Zones. No internet gateway route, and for a truly isolated build, no NAT gateway either. Create one security group for SageMaker resources with a self-referencing inbound rule on all TCP ports:

SG=$(aws ec2 create-security-group --group-name sagemaker-private \
  --description "SageMaker AI private" --vpc-id vpc-0abc123 \
  --query GroupId --output text)

aws ec2 authorize-security-group-ingress --group-id $SG \
  --protocol tcp --port 0-65535 --source-group $SG

The self-reference is not optional: Studio apps, distributed training containers, and the interface endpoint ENIs all talk to each other through this group. Missing it is the most common cause of a Studio space that spins forever on "Starting."

Step 2: The VPC endpoints you actually need

With no NAT, every AWS API call needs a private path. The minimum working set for training and real-time inference:

Service name	Type	Why
`com.amazonaws.<region>.sagemaker.api`	Interface	Control plane: create/describe jobs
`com.amazonaws.<region>.sagemaker.runtime`	Interface	`InvokeEndpoint` calls
`aws.sagemaker.<region>.studio`	Interface	Studio / Unified Studio app traffic
`com.amazonaws.<region>.sts`	Interface	Role assumption inside containers
`com.amazonaws.<region>.ecr.api` and `.ecr.dkr`	Interface	Pulling the training/serving image
`com.amazonaws.<region>.logs`	Interface	CloudWatch Logs from jobs
`com.amazonaws.<region>.monitoring`	Interface	Endpoint and job metrics
`com.amazonaws.<region>.s3`	Gateway (plus interface, see below)	Data, artifacts, and ECR layer storage
`com.amazonaws.<region>.kms`	Interface	Envelope encryption calls

Add sagemaker.featurestore-runtime if you read features online, and codeartifact.api / codeartifact.repositories if your containers pip install at runtime.

S3 deserves a note. A gateway endpoint is free and covers traffic originating in the VPC, which includes ECR image layers — image pulls fail without it. Studio's file system and some job paths, however, resolve S3 from outside your subnets, so many teams add an interface S3 endpoint as well and keep both. If you add the interface endpoint, disable its private DNS or use the endpoint-specific hostnames, or you will shadow the gateway route.

aws ec2 create-vpc-endpoint --vpc-id vpc-0abc123 \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.us-east-1.sagemaker.api \
  --subnet-ids subnet-0aaa subnet-0bbb \
  --security-group-ids $SG --private-dns-enabled

Budget for this: interface endpoints bill roughly $0.01 per hour per AZ per endpoint plus data processing. Nine endpoints across two AZs is on the order of $130–$150 a month — usually far less than the NAT gateway it replaces.

Step 3: A VPC-only domain

aws sagemaker create-domain \
  --domain-name analytics-private \
  --auth-mode IAM \
  --vpc-id vpc-0abc123 \
  --subnet-ids subnet-0aaa subnet-0bbb \
  --app-network-access-type VpcOnly \
  --kms-key-id arn:aws:kms:us-east-1:111122223333:key/<key-id> \
  --default-user-settings '{
    "ExecutionRole":"arn:aws:iam::111122223333:role/SageMakerStudioPrivate",
    "SecurityGroups":["'"$SG"'"]
  }'

VpcOnly is the whole point: the app ENIs land in your subnets and there is no AWS-managed egress path. Note that AppNetworkAccessType cannot be changed on a domain with running apps, so decide before you onboard users.

Two consequences to plan for. Users lose pip install from PyPI and git clone from GitHub — mirror what they need in CodeArtifact and CodeCommit or an internal Git host behind PrivateLink. And JumpStart model downloads need an S3 path; JumpStart works in VpcOnly domains but only with the S3 endpoints in place.

Step 4: Jobs inside the VPC, with network isolation

Placement is per-job, not inherited from the domain. In the Python SDK:

from sagemaker.estimator import Estimator
from sagemaker.network import NetworkConfig

net = NetworkConfig(
    enable_network_isolation=True,
    security_group_ids=["sg-0abc123"],
    subnets=["subnet-0aaa", "subnet-0bbb"],
    encrypt_inter_container_traffic=True,
)

est = Estimator(
    image_uri=training_image,
    role=role,
    instance_type="ml.g5.2xlarge",
    instance_count=2,
    subnets=net.subnets,
    security_group_ids=net.security_group_ids,
    enable_network_isolation=True,
    encrypt_inter_container_traffic=True,
    volume_kms_key="arn:aws:kms:us-east-1:111122223333:key/<key-id>",
    output_kms_key="arn:aws:kms:us-east-1:111122223333:key/<key-id>",
    output_path="s3://ml-artifacts-private/training/",
)
est.fit({"train": "s3://ml-data-private/train/"})

enable_network_isolation=True goes further than VPC placement: the container itself gets no network access at all. SageMaker copies input data in and artifacts out on your behalf. That means code inside the job cannot call S3, cannot download a checkpoint from the Hub, and cannot phone a license server — all inputs must arrive as channels. It is the strongest control available and worth the extra plumbing for anything touching regulated data. (It is incompatible with jobs that must reach the internet, and it cannot be used with Local Mode.)

encrypt_inter_container_traffic=True matters only for distributed training, and it does add overhead on multi-node jobs; measure before assuming it is free.

The same three settings apply to processing jobs (NetworkConfig on Processor) and to models:

from sagemaker.model import Model

model = Model(
    image_uri=serving_image,
    model_data=est.model_data,
    role=role,
    vpc_config={"Subnets": ["subnet-0aaa", "subnet-0bbb"],
                "SecurityGroupIds": ["sg-0abc123"]},
    enable_network_isolation=True,
)
predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.m6i.xlarge",
    kms_key="arn:aws:kms:us-east-1:111122223333:key/<key-id>",
)

Step 5: Keys and bucket policy

Use a customer-managed key, not aws/sagemaker, so you keep the ability to audit and revoke. The key policy must allow the execution role kms:Encrypt, Decrypt, GenerateDataKey*, DescribeKey, and — for EBS volumes — CreateGrant. Forgetting CreateGrant produces a training job that fails immediately with an opaque AccessDenied on volume creation.

Then make the data bucket refuse anything that did not come through your endpoint:

{
  "Sid": "DenyNonVpceAccess",
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": ["arn:aws:s3:::ml-data-private",
               "arn:aws:s3:::ml-data-private/*"],
  "Condition": {
    "StringNotEquals": {"aws:sourceVpce": "vpce-0abc123"}
  }
}

Attach this only after the endpoints work, and keep a break-glass admin principal excluded from the deny — a StringNotEquals on aws:PrincipalArn in the same condition block — or you will lock yourself out of your own bucket.

Step 6: Make it the only option

Configuration drifts. IAM conditions do not. Attach a policy to the execution role (or an SCP to the account) that denies job creation outside the approved network:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RequireVpcAndIsolation",
      "Effect": "Deny",
      "Action": [
        "sagemaker:CreateTrainingJob",
        "sagemaker:CreateProcessingJob",
        "sagemaker:CreateModel"
      ],
      "Resource": "*",
      "Condition": {
        "Null": {"sagemaker:VpcSubnets": "true"}
      }
    },
    {
      "Sid": "RequireNetworkIsolation",
      "Effect": "Deny",
      "Action": ["sagemaker:CreateTrainingJob", "sagemaker:CreateModel"],
      "Resource": "*",
      "Condition": {
        "Bool": {"sagemaker:NetworkIsolation": "false"}
      }
    },
    {
      "Sid": "RequireCmk",
      "Effect": "Deny",
      "Action": "sagemaker:CreateTrainingJob",
      "Resource": "*",
      "Condition": {
        "Null": {"sagemaker:VolumeKmsKey": "true"}
      }
    }
  ]
}

sagemaker:VpcSubnets, sagemaker:VpcSecurityGroupIds, sagemaker:NetworkIsolation, sagemaker:InterContainerTrafficEncryption, and sagemaker:VolumeKmsKey are the condition keys that carry the weight here. With these in place, a data scientist who forgets NetworkConfig gets a clear denial instead of a silent policy violation.

Verifying it, not assuming it

Four checks we run before signing off an environment:

Egress test. From a Studio terminal: curl -m 5 https://example.com. It must time out. If it returns a page, the domain is not VpcOnly or a route to a NAT survives.
Endpoint proof. In a notebook, import socket; socket.gethostbyname("sagemaker.us-east-1.amazonaws.com") should resolve to a private RFC 1918 address from your subnet CIDR.
Flow logs. Enable VPC Flow Logs and confirm training-job ENIs talk only to endpoint ENIs and S3 prefix lists.
CloudTrail on KMS. kms:Decrypt events should name your execution role and your key, nothing else.

Failure modes and what they mean

Job sits in Starting, then ResourceInitializationError or an image pull failure → missing ecr.api, ecr.dkr, or the S3 gateway endpoint.
Studio app never leaves Pending → security group has no self-referencing rule, or aws.sagemaker.<region>.studio endpoint is missing.
AccessDenied seconds into a job → KMS key policy missing CreateGrant for the execution role.
pip hangs for 120 seconds and dies → expected in VpcOnly; point users at CodeArtifact.
Endpoint deploys but InvokeEndpoint from another VPC fails → the caller needs its own sagemaker.runtime endpoint or cross-VPC routing.

Where this lands

Done properly, this is a two- to three-day build for a single account and a week or so with landing-zone patterns, multiple accounts, and a CDK or Terraform module to make it repeatable. The payoff is that the security review becomes a walkthrough instead of a negotiation, and every future model inherits the controls rather than re-litigating them.

If you are standing up SageMaker AI inside compliance constraints — HIPAA, PCI, FedRAMP-aligned, or an internal policy that simply says no data on the public internet — our consultants have built this pattern across regulated environments. Get in touch with your VPC layout and constraints and we will tell you what it takes.