SageMaker Edge Manager was the AWS answer to "how do I run my SageMaker model on a device" until it was discontinued on April 26, 2024. AWS's stated replacement is ONNX for a portable runtime plus AWS IoT Greengrass V2 for deployment, which, in practice, is a better stack than Edge Manager ever was. This tutorial builds it on an NVIDIA Jetson Orin: export a PyTorch model trained on SageMaker AI to ONNX, package inference as a Greengrass component, deploy it over the air, and update the model later without touching the code.
Hardware assumptions: a Jetson Orin Nano or AGX Orin running JetPack 6 (Ubuntu 22.04, CUDA 12), with network access. Everything also works on a Raspberry Pi 5 with the CPU execution provider; only the runtime wheel changes.
1. Export the model to ONNX
Start from the model.tar.gz your SageMaker training job wrote to S3. In a Studio space or locally:
import torch, torchvision
model = torchvision.models.resnet50()
model.load_state_dict(torch.load("model/resnet50.pt", map_location="cpu"))
model.eval()
dummy = torch.randn(1, 3, 224, 224)
torch.onnx.export(
model, dummy, "resnet50.onnx",
input_names=["input"], output_names=["logits"],
dynamic_axes={"input": {0: "batch"}, "logits": {0: "batch"}},
opset_version=18,
dynamo=True,
)
dynamo=True uses the current TorchDynamo-based exporter (the default in recent PyTorch releases); drop it if you are on an older PyTorch. Then verify the export reproduces the PyTorch output before anything goes near a device:
import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("resnet50.onnx", providers=["CPUExecutionProvider"])
onnx_out = sess.run(None, {"input": dummy.numpy()})[0]
torch_out = model(dummy).detach().numpy()
print("max abs diff:", np.abs(onnx_out - torch_out).max()) # expect ~1e-5
Upload the validated file to S3 with a version in the key, for example s3://my-edge-models/resnet50/1.0.0/resnet50.onnx. Greengrass components reference artifacts by S3 URI, and versioned keys are what make rollbacks trivial later.
For best Jetson performance you can additionally let ONNX Runtime's TensorRT execution provider build an engine on first run; keep the ONNX file as the source of truth, since the TensorRT engine is specific to one GPU and JetPack version.
2. Install Greengrass V2 on the device
On the Jetson, with AWS credentials that can create the thing and role (use a short-lived session, not a long-lived key):
sudo apt-get update && sudo apt-get install -y default-jre python3-pip
curl -s https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-nucleus-latest.zip \
-o greengrass-nucleus-latest.zip
unzip greengrass-nucleus-latest.zip -d GreengrassInstaller
sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE \
-jar ./GreengrassInstaller/lib/Greengrass.jar \
--aws-region us-east-1 \
--thing-name orin-line1-cam3 \
--thing-group-name edge-inference-fleet \
--component-default-user ggc_user:ggc_group \
--provision true --setup-system-service true
The installer registers the device as an IoT thing, creates a token-exchange role, and starts the nucleus as a systemd service. The thing group (edge-inference-fleet) is how you will deploy to many devices at once. Full details are in the IoT Greengrass V2 developer guide.
Install ONNX Runtime for the device. On Jetson use the GPU wheel that NVIDIA publishes for your JetPack release; on a Pi, pip3 install onnxruntime is enough.
3. Write the inference component
A Greengrass component is a recipe (YAML or JSON) plus artifacts. Keep the model in its own component so it can be versioned independently of the code.
Model component, com.example.ResNet50Model (recipe only; the artifact is the ONNX file in S3):
RecipeFormatVersion: "2020-01-25"
ComponentName: com.example.ResNet50Model
ComponentVersion: "1.0.0"
ComponentDescription: ResNet-50 ONNX model exported from SageMaker AI
ComponentPublisher: NeuralArmada
Manifests:
- Platform:
os: linux
Artifacts:
- URI: s3://my-edge-models/resnet50/1.0.0/resnet50.onnx
Inference component, com.example.EdgeInference, with a small Python service that watches a folder (or a camera) and publishes results to IoT Core:
# inference.py
import json, os, sys, time, glob
import numpy as np
import onnxruntime as ort
from PIL import Image
from awsiot.greengrasscoreipc.clientv2 import GreengrassCoreIPCClientV2
MODEL_PATH = sys.argv[1]
THING = os.environ["AWS_IOT_THING_NAME"]
TOPIC = f"edge/{THING}/inference"
providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]
sess = ort.InferenceSession(MODEL_PATH, providers=providers)
ipc = GreengrassCoreIPCClientV2()
MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
def preprocess(path):
img = Image.open(path).convert("RGB").resize((224, 224))
x = (np.asarray(img, dtype=np.float32) / 255.0 - MEAN) / STD
return x.transpose(2, 0, 1)[None]
while True:
for path in glob.glob("/var/edge/inbox/*.jpg"):
t0 = time.time()
logits = sess.run(None, {"input": preprocess(path)})[0][0]
top = int(np.argmax(logits))
ipc.publish_to_iot_core(
topic_name=TOPIC, qos="1",
payload=json.dumps({
"file": os.path.basename(path), "class": top,
"score": float(np.max(logits)), "latency_ms": (time.time() - t0) * 1000,
"model": os.path.basename(MODEL_PATH),
}).encode(),
)
os.remove(path)
time.sleep(0.2)
Its recipe declares a dependency on the model component and passes the artifact path in:
RecipeFormatVersion: "2020-01-25"
ComponentName: com.example.EdgeInference
ComponentVersion: "1.0.0"
ComponentPublisher: NeuralArmada
ComponentDependencies:
com.example.ResNet50Model:
VersionRequirement: ">=1.0.0 <2.0.0"
DependencyType: HARD
ComponentConfiguration:
DefaultConfiguration:
accessControl:
aws.greengrass.ipc.mqttproxy:
com.example.EdgeInference:mqttproxy:1:
policyDescription: Publish inference results
operations:
- aws.greengrass#PublishToIoTCore
resources:
- "edge/*/inference"
Manifests:
- Platform:
os: linux
Lifecycle:
Install: pip3 install --user awsiotsdk pillow numpy
Run: >
python3 {artifacts:path}/inference.py
{com.example.ResNet50Model:artifacts:path}/resnet50.onnx
Artifacts:
- URI: s3://my-edge-components/edge-inference/1.0.0/inference.py
The accessControl block is the part people forget: without it the IPC publish call is denied. HARD dependency means the inference component restarts when the model component changes, which is exactly the behaviour you want for model updates.
4. Publish the components and deploy to the fleet
aws greengrassv2 create-component-version --inline-recipe fileb://model-recipe.yaml
aws greengrassv2 create-component-version --inline-recipe fileb://inference-recipe.yaml
aws greengrassv2 create-deployment \
--target-arn arn:aws:iot:us-east-1:123456789012:thinggroup/edge-inference-fleet \
--deployment-name edge-inference-v1 \
--components '{
"com.example.ResNet50Model": {"componentVersion": "1.0.0"},
"com.example.EdgeInference": {"componentVersion": "1.0.0"}
}'
Every device in the group pulls the artifacts from S3 (the token-exchange role needs s3:GetObject on both buckets), runs the lifecycle, and reports status. Watch it with aws greengrassv2 list-effective-deployments --core-device-thing-name orin-line1-cam3, and on the device with sudo tail -f /greengrass/v2/logs/com.example.EdgeInference.log. Drop a JPEG into /var/edge/inbox/ and subscribe to edge/+/inference in the IoT Core test client to see results arrive.
5. Over-the-air model updates
Retrain on SageMaker AI, re-export, validate the ONNX output exactly as in step 1, upload to s3://my-edge-models/resnet50/1.1.0/resnet50.onnx, and publish the model component as version 1.1.0 with the new URI. Then a new deployment:
aws greengrassv2 create-deployment \
--target-arn arn:aws:iot:us-east-1:123456789012:thinggroup/edge-inference-fleet \
--deployment-name edge-inference-model-1-1-0 \
--components '{
"com.example.ResNet50Model": {"componentVersion": "1.1.0"},
"com.example.EdgeInference": {"componentVersion": "1.0.0"}
}' \
--deployment-policies '{"failureHandlingPolicy": "ROLLBACK"}' \
--iot-job-configuration '{"jobExecutionsRolloutConfig": {"maximumPerMinute": 10}}'
The inference code never changed, so only the model artifact downloads. ROLLBACK returns a device to the previous component set if the new one fails its lifecycle, and the rollout rate limits how many devices update per minute, so a bad model cannot take down a whole line at once. Rolling back deliberately is the same command with 1.0.0.
Monitoring the fleet
Edge Manager's fleet dashboard is gone; its replacement is a combination of Greengrass deployment status (per-device component health), the inference results you publish to IoT Core (route them to CloudWatch metrics or Timestream with an IoT rule for latency and class distributions), and the aws.greengrass.LogManager and aws.greengrass.TelemetryEmitter public components for logs and device telemetry. Ship a small sample of inputs back to S3 for drift analysis against your training set.
This is the reference architecture we use in edge modernization engagements; if you are still on Edge Manager or Greengrass V1, get in touch.