Amazon SageMaker Edge Manager was discontinued on April 26, 2024. If you built edge ML on AWS between 2020 and 2023, there is a reasonable chance your fleet management, model packaging, and device agent all depended on it. Two years on, this post describes what AWS recommends instead, the reference architecture we deploy, the hardware that makes sense now, and how to monitor a fleet without the dashboard you lost.
What Edge Manager did, and what replaced it
Edge Manager packaged a model (usually after SageMaker Neo compilation), signed it, delivered it to devices through IoT Greengrass, ran it on-device through a dedicated agent, and reported fleet status and sample inference data back to the console. Four jobs in one service.
AWS's end-of-life guidance replaces it with two general-purpose pieces:
- ONNX (and ONNX Runtime) for a portable model format and cross-platform inference runtime.
- AWS IoT Greengrass V2 for packaging, deployment, over-the-air updates, and device-side orchestration.
Fleet monitoring, the fourth job, is assembled from Greengrass telemetry, IoT Core, and CloudWatch. SageMaker Neo still exists for compilation, but for most fleets an ONNX Runtime execution provider (TensorRT on NVIDIA, OpenVINO on Intel, CPU everywhere) gets the performance without locking the artifact to one compiler.
This is, frankly, a better design. Edge Manager's agent was opaque and its model format was its own; ONNX plus Greengrass components are open, inspectable, and usable for non-ML workloads on the same device.
The reference architecture
SageMaker AI training job
| export + validate
v
ONNX model in S3 (versioned key)
| Greengrass component (model) <-- independent version
v
Greengrass component (inference code) --> IoT Core topic --> IoT rule --> CloudWatch / Timestream / S3
|
v
Device: Greengrass nucleus + ONNX Runtime (+ TensorRT / OpenVINO EP)
The design choices that matter:
- Two components, not one. The model and the inference code are separate Greengrass components with a
HARDdependency. Model updates then touch only the model artifact, roll out at a controlled rate, and roll back independently. This one decision removes most of the pain teams had with monolithic Edge Manager packages. - Validation before packaging. Every exported ONNX model is checked against the PyTorch output on a fixed input set, and against the previous model on a held-out sample, before its S3 key is referenced by a component version. Nothing unvalidated gets a version number.
- Versioned S3 keys and semantic component versions. Rollback is a deployment with an older version, nothing more.
- Results published as structured messages. Each inference publishes a small JSON record (class, score, latency, model version, device) to an IoT Core topic. An IoT rule fans it out to CloudWatch metrics for dashboards and alarms, and to S3 or Timestream for analysis.
- Staged rollouts through thing groups. A canary group of a few devices gets every deployment first; the fleet follows after the canary metrics hold for a set window.
We walk through the implementation, recipes and code included, in our tutorial on edge inference with ONNX Runtime and Greengrass V2 on Jetson Orin.
Hardware in 2026
The device landscape moved while Edge Manager was being retired:
- NVIDIA Jetson Orin (Nano, NX, AGX) is the default for GPU inference at the edge. Jetson TX2 and the original Nano are end-of-life; JetPack 6 on Orin gives you CUDA 12 and a supported ONNX Runtime with the TensorRT execution provider. The newer Jetson Thor targets robotics and heavier workloads.
- Raspberry Pi 5 is a credible CPU inference target for small models (quantized vision and audio models, tabular models), and an inexpensive way to run a fleet pilot.
- Intel and AMD industrial boards run ONNX Runtime with the OpenVINO or default CPU providers; AMD's acquisition of Xilinx means the FPGA story is now AMD's, with Vitis AI consuming ONNX.
- Arm Cortex-A gateways handle lightweight models; for microcontrollers, ONNX is usually converted further (for example to TensorFlow Lite Micro or a vendor runtime) and Greengrass runs on a nearby gateway rather than the MCU.
If you are still on TX2, plan the hardware refresh in the same project as the software migration; the driver and runtime support you need is not coming to the old boards.
Fleet monitoring options
Edge Manager's dashboard showed device status and sampled inference data. Rebuild those two functions deliberately:
Device and component health. Greengrass deployment status (per device, per component) is available in the console and through ListEffectiveDeployments. The aws.greengrass.TelemetryEmitter public component publishes CPU, memory, and component-state telemetry; aws.greengrass.LogManager ships component logs to CloudWatch Logs. Alarm on a device that has not reported in N minutes and on component restarts.
Inference quality. From the structured result messages: per-device latency percentiles, throughput, and class distribution as CloudWatch metrics. A shift in the class distribution on one camera, or a latency jump after a model update, is the drift signal Edge Manager used to sample for you. Ship a small random sample of raw inputs to S3 (with the model version in the key) so you can evaluate the current model against reality and build the next training set.
Model versions in the field. Every result message carries the model version. A CloudWatch metric dimensioned by version tells you exactly how far a rollout has gone and whether the new version behaves differently.
Migration path from Edge Manager
- Export your existing models to ONNX from the framework they were trained in (PyTorch, TensorFlow, or, if you are also on a retired framework, after porting). Validate numerically.
- Upgrade devices to Greengrass V2 if they are on V1; V1 reached end of support in 2023. Provision into thing groups that reflect your rollout stages.
- Replace the Edge Manager agent with an inference component that loads ONNX Runtime and publishes results to IoT Core. The tutorial above is a working starting point.
- Rebuild monitoring from the telemetry and result streams.
- Decommission Edge Manager packaging jobs and device fleets in the SageMaker console, and remove the agent from device images.
A fleet of a few dozen devices is typically a four-to-six-week project including the validation windows. Larger fleets are mostly about rollout discipline.
Edge ML on AWS in 2026 is in better shape than it was with Edge Manager; it just asks you to assemble the pieces. If you would rather not assemble them alone, see our legacy ML modernization page or contact us.