Autoscaling that looks ahead.
Machine Learning Autoscaler forecasts the next minute of traffic with a random forest and starts serverless pods before the thirty-second cold start can hurt.
- Reactive
- Predictive
- Requests slower than 300 ms
- –
- –
- p95 latency
- –
- –
- Pod-minutes used
- –
- –
- –fewer slow requests when a sale starts
- –requests per second: the typical forecast error in the sale pattern
- –less pod-time than provisioning for the peak, in the sale pattern
What this is
Serverless platforms add copies of a function when traffic arrives, and each copy needs about thirty seconds before it can serve anyone. This project tests whether a random forest can forecast the next minute of load, so copies start early.
The problem. A reactive autoscaler measures load, then adds pods. For the thirty seconds the new pods take to start, the old ones are overloaded: latency climbs and requests are dropped. A ticket sale or a scheduled job makes that gap visible to every user at once.
The approach. A scikit-learn random forest reads the last minute of request rates from Prometheus, plus the position in the daily cycle, and predicts the peak load of the next sixty seconds. The scaler sizes for the larger of that forecast and the live load.
The stack. Python functions run on OpenFaaS in a kind cluster on Kubernetes. Prometheus records request rate, latency and replica counts, and Grafana draws them. hey generates the load. One Flask service holds the model and the control loop.
The caveat. The numbers on this site come from a simulator fed synthetic traffic, running the same scaling code as the live service. They show the mechanism, not a production result. The deploy guide lets you measure your own cluster.