There is a particular kind of production ML system that still works perfectly and is nonetheless a liability: the model was trained in 2018 with Apache MXNet or Chainer, it runs in a container that has not been rebuilt since, and nobody on the current team has opened the training code. This post is the playbook we use to move those systems to PyTorch on SageMaker AI: the timeline that got us here, the actual risks, the conversion strategy, and the validation harness that makes the switch safe.
The timeline
- December 2019: Preferred Networks ends Chainer development and moves to PyTorch. Chainer enters maintenance-only mode; there have been no feature releases since.
- 2020 onward: Amazon, which had backed MXNet as its deep learning framework of choice and built Gluon on it, shifts its own research and product work to PyTorch.
- September 2023: the Apache Software Foundation retires MXNet to the Apache Attic. No releases, no security fixes, no mailing list. Gluon, being MXNet's high-level API, goes with it.
- 2024 to 2026: SageMaker AI continues to ship current containers for PyTorch, TensorFlow, Hugging Face, and XGBoost. MXNet and Chainer containers are frozen at their last versions.
None of this made your model worse. It made the ground under it unsupported.
The actual risks
Frozen containers accumulate CVEs. The last MXNet deep learning containers carry the Python, CUDA, and system libraries of their era. Every vulnerability scanner your security team runs will flag them, and there is no upstream fix coming. The same applies to a Chainer image you built yourself.
Hardware and driver drift. New SageMaker instance families ship with newer CUDA drivers. An old framework build will eventually fail to load on the instance types that remain available, usually at the worst possible moment.
Hiring and bus factor. Nobody is learning MXNet or Chainer in 2026. The engineer who can read your training script is the only one, and the model is effectively unreproducible once they leave.
Tooling exclusion. Current SageMaker features (Pipelines conveniences, Model Monitor integrations, JumpStart, the LMI serving stack, Inferentia compilation) assume a living framework. Every new capability you want becomes a workaround.
Conversion strategy
Step 1: inventory and reproduce
List every MXNet / Chainer model: training code, data sources, the container image, where it is deployed, and who owns it. Then try to reproduce the original training run on the original stack. If you cannot, you have found your first problem, and the conversion plan changes (see "when to retrain" below).
Step 2: choose convert or retrain
There are two routes, and the inventory tells you which:
- Convert the architecture and port the weights. Write the model in PyTorch, map layer by layer, and copy the trained parameters across. Works well for standard architectures (ResNets, LSTMs, straightforward MLPs) and when retraining is expensive or the original data is gone. Gluon models port cleanly because the block structure maps almost one-to-one to
torch.nn.Module; Chainer'sChain/Linkmodel is similar. - Retrain from scratch in PyTorch. Simpler when the model is small, the data is available, and the original training is reproducible. Use a current architecture if the old one was a compromise for its time; a 2018 image classifier can often be replaced by a fine-tuned modern backbone with fewer parameters and better accuracy.
ONNX is a useful intermediate for some MXNet models: export from MXNet to ONNX, import into PyTorch or serve the ONNX directly. In practice the exporters have gaps (custom operators, some RNN variants), so treat it as a shortcut to try, not a plan.
Step 3: port, with a numerical harness
Whatever route you take, build the validation harness before you write the new model. It has three levels:
- Layer-level parity. For a ported model, feed identical inputs to the old and new implementations and compare activations at each layer. Differences should be at floating-point noise (1e-5 or so); anything larger means a transposed weight, a padding convention, or a different default (MXNet's
NCHWversus other layouts, BatchNorm momentum conventions, Chainer'sLinearweight orientation). - Prediction parity. On a held-out set of a few thousand records, compare outputs. For classification: agreement rate and confusion between the two models. For regression: distribution of the difference. Set a threshold in advance and do not move it afterwards.
- Business-metric parity. Whatever the model is for (conversion, fraud catch rate, forecast error), measure both models on the most recent month of real data. This is the number the stakeholders will ask about.
Check the harness into the repository and run it in CI against both models until the old one is decommissioned.
Step 4: rebuild the serving path on current containers
Deploy the PyTorch model to SageMaker AI on a current PyTorch or Hugging Face deep learning container. Use the same endpoint contract (request and response schema) so callers do not change. Put data capture on, and shadow-test the new endpoint with a copy of production traffic for a week before cutting over; SageMaker's shadow variants make this a configuration change rather than an application change.
Step 5: cut over and decommission
Switch traffic, keep the old endpoint for a defined rollback window, then delete it, the container image, and the notebook instance it lived on. Migration is not done until the old stack is gone and the security scanner stops complaining.
A representative engagement
A typical case we see: a recommendation model, Gluon, trained in 2019, serving a few hundred requests per second from a pair of ml.c5 instances behind an application the company still depends on. Training data is in S3 and the training script still runs. Conversion is a port (the architecture is a two-tower model with embedding layers and a small MLP), the numerical harness passes at the first level within a day, and prediction parity is above 99.9 percent agreement. The new endpoint runs on a current PyTorch container, on Graviton instances for less than the original cost, and the shadow week shows identical business metrics. The old endpoint goes away a month later. Elapsed time, inventory to decommission: about five weeks, most of it waiting on the shadow window and change approvals rather than engineering.
The hard cases are the ones where step 1 fails: the training data is partially gone, the script depends on a preprocessing job nobody can find, and the model can only be validated by its outputs. Those become a retrain-with-parity project, and the timeline doubles.
Where to start
Run the inventory. Most teams are surprised by how many frozen containers they are actually running, and the list alone is enough to get the migration budgeted. If you would rather we did it, our legacy ML modernization page describes the engagement, and we are happy to talk it through.