What survives when a model learns something new?
A real, executed continual-learning benchmark. Five tasks arrive one after another: handwritten digits 0–1, 2–3, 4–5, 6–7, then 8–9. We compare updating model weights alone, replaying old experiences while updating, and retaining a small exemplar memory instead. All plotted numbers come from three executed seeds, not illustrative slider values.
Actual measured learning curves
Choose a metric. Values are the mean of seeds 7, 13, and 29 evaluated on held-out human-written digits after each task.
Final checkpoint, after all five tasks
| Method | All seen | First task (0–1) | Last task (8–9) |
|---|---|---|---|
| Weight updates only | 18.96% | 0.00% | 96.97% |
| Reservoir replay | 91.33% | 81.32% | 94.70% |
| Exemplar memory | 92.96% | 99.27% | 79.92% |
Real provenance. Reproducible source.
The 1,797 images are from the public UCI Optical Recognition of Handwritten Digits dataset loaded via sklearn.datasets.load_digits. We used an untouched 25% stratified test split per seed, 120 equal-count SGD minibatch updates per stage (32 samples/batch), and at most 160 stored real training samples. No mock, synthetic, or handmade benchmark records.
Run the experiment · Inspect measured results JSON
Limitations: This is a small real image-classification probe, not an LLM or coding-agent evaluation; the replay and k-nearest-neighbor methods do not have matched inference cost. Only three random seeds and one hyperparameter setting were tested. Read the full methods and limitations.