Machine learning engineering, MLOps
Production ML Platform Modernisation
Rebuilding hand operated model workflows into managed pipelines, cutting manual effort and infrastructure spend at the same time.
- Role
- Machine Learning Engineer
- Organisation
- Enterprise engagement
- Period
- 2022 to 2024
- Status
- Shipped
- Domain
- Machine learning engineering, MLOps
01
Overview
A data science team was producing good models and struggling to keep them running. Training happened in notebooks on individual machines, deployment was a manual handover, and nobody could reliably say which version of which model was serving production traffic.
The work was to turn that into a platform: reproducible training, automated deployment, monitored inference, and an infrastructure bill that reflected actual usage.
02
Problem
Every retrain was a person following a runbook. That made the schedule dependent on availability, made reproducing a past result close to impossible, and put a ceiling on how many models the team could support.
The cost profile was equally unmanaged. Training instances stayed up between jobs, inference endpoints were provisioned for a peak that arrived twice a month, and development environments ran continuously because nobody owned turning them off.
03
Solution
Training was moved into managed pipelines with versioned inputs, versioned artefacts, and a registry that records exactly which dataset and code produced each model. Retraining became a triggered pipeline rather than an afternoon.
Deployment was automated behind approval gates, with monitoring on input distributions and output quality so a degrading model surfaces before a stakeholder notices. Cost work ran in parallel: right sized instances, scheduled shutdown of idle development capacity, spot capacity for training, and autoscaling inference sized to real traffic rather than the worst case.
04
Technical architecture
- 01
Data and features
Versioned datasets in object storage with schema validation at the pipeline boundary, so a malformed input fails the run rather than the model.
- 02
Training
SageMaker pipelines defined in code, running on spot capacity, emitting metrics and artefacts to a registry with full lineage back to dataset and commit.
- 03
Evaluation gate
Automated comparison against the incumbent model on a held out set. A candidate that does not clear the bar cannot be promoted.
- 04
Deployment
Registry promotion triggers a deployment pipeline with staged rollout, so a regression is caught on a slice of traffic rather than all of it.
- 05
Serving
Autoscaling endpoints sized from observed traffic, with request and response logging feeding the monitoring layer.
- 06
Monitoring
Input drift and output quality tracked continuously, with alerting that names the affected model and the metric that moved.
05
Technologies
Platform
- AWS SageMaker
- S3
- Lambda
- CloudWatch
Infrastructure as code
- Terraform
- Docker
- GitHub Actions
Modelling
- Python
- scikit-learn
- PyTorch
- MLflow
Practice
- Model registry
- Drift monitoring
- Cost optimisation
06
Challenges
Migrating without stopping delivery
The existing models had to keep serving throughout. The migration ran model by model, with the old path live until the new pipeline had reproduced its results, which was slower to plan and considerably safer.
Reproducibility was not just a tooling gap
Several models could not be reproduced at all because the exact training data no longer existed. Establishing dataset versioning first, before any pipeline work, was the only way to make the rest meaningful.
Cutting cost without cutting capability
Spot capacity introduces interruption, and autoscaling introduces cold starts. Both needed handling in the pipeline rather than accepted as a quality cost, so the saving did not simply move the problem to the users.
07
Outcomes
- 90%Reduction in manual effort per model retrain and deployment
- £100kAnnual infrastructure spend removed with no reduction in delivery pace
- Full lineage from serving model back to dataset and commit
- Model degradation detected by monitoring rather than by stakeholders
08
Key learnings
- Reproducibility is the prerequisite for everything else. Automating a process you cannot reproduce only makes the uncertainty faster.
- Cost work is engineering work. The savings came from understanding the workload, not from negotiating a discount.
- A platform succeeds when the team stops asking permission to retrain. Adoption is the metric that matters, not architecture.