What survives when a model learns something new?

TL;DR

A model can learn a new task and, in the process, lose something it previously knew. We tested this under a simple, controlled sequence of five classification tasks built from 1,797 real handwritten digit images. An online classifier trained only on the newest task retained just 19.0% accuracy across all previously seen digit classes at the end. Keeping a small memory of earlier examples changed the outcome: rehearsal achieved 91.3%, and a retrieval-only memory achieved 93.0%. These are measured averages across three seeded runs, not illustrative values.

The result is useful as a concrete illustration of catastrophic forgetting. It is not evidence that the same approaches improve LLMs or coding agents. The purpose here was to examine the retention–adaptation tradeoff before attempting an agent-scale evaluation.

1. The question

Suppose a system works well today, encounters a new task tomorrow, and changes how it behaves. That change might live in its model weights, in a replay buffer used during training, or in an external memory consulted at inference time. All three can help it adapt, but they preserve old abilities in different ways.

The difficulty is that performance on the latest task is not a reliable measure of continual learning. A system might score highly on its new task while failing at an old one. To see that failure rather than hide it, we need to evaluate earlier tasks again after each new training stage.

For this first experiment, we deliberately use a small real-world classification setting instead of an expensive agent benchmark. This isolates the question of retention; it does not pretend that classifying digits captures every complication of learning from agent trajectories.

2. Experimental setup

2.1 Dataset and task sequence

The data are the UCI Optical Recognition of Handwritten Digits images, loaded through sklearn.datasets.load_digits(). There are 1,797 authentic handwritten examples, each represented by 64 pixel intensities. No synthetic or handwritten-for-this-demo evaluation cases were added.

Each seed has a stratified 75% training / 25% held-out testing split. The learner sees five tasks in this fixed order:

2.2 Three ways to handle past experience

Weights only. A multiclass logistic classifier receives 120 stochastic-gradient minibatch updates per stage, using only examples of the two most recent classes. It has no way to revisit earlier examples. This is our forgetting-prone baseline.

Reservoir replay. The classifier gets the same 120 parameter-update steps per stage, but can fill half a minibatch from an online reservoir of at most 160 previously encountered training examples. The reservoir samples uniformly from experience seen so far. Learning still happens in model weights, while memory supplies rehearsal data.

Exemplar memory. A distance-weighted, 3-nearest-neighbor classifier predicts labels using a reservoir of 160 real training examples. It does not use the same parameter-update process. This makes it a useful retrieval-only reference, but not an equal-compute comparison to the other models.

We used seeds 7, 13, and 29 and evaluated every method on that seed’s untouched test split. The full Python implementation specifies the data, sampling, training loop, and evaluation procedure.

3. Findings

3.1 Learning the new task is not retaining the old ones

At the final stage, the weight-only model is still good at classifying the new digits, but its accuracy on the first task has dropped to 0%. This is the central failure the experiment was meant to expose: a high latest-task score can coexist with almost complete loss of earlier ability.

3.2 Memory changes the retention–adaptation balance

The replay model preserves much more of the first task while learning the final task reasonably well. The retrieval-only system keeps the earliest task almost intact, but scores lower on the newest one. These numbers should not be used to declare a universal winner: the methods have different computational costs and inductive biases.

4. Interpretation

A growing system has at least two separate obligations: acquire new behavior and avoid unnecessary regressions. The baseline optimizes the first with almost no protection for the second. Replay reintroduces earlier data during adaptation; exemplar retrieval avoids overwriting a learned classifier altogether.

This is a useful reminder for work on continual learning and agent memory, where it is tempting to measure only performance on the latest benchmark. A more complete evaluation should ask what happens to older tasks, whether retained information transfers to unseen situations, and what it costs in compute, storage, and inference time.

My separate coding-agent memory study examines another piece of this problem: whether earlier experiences should be promoted and exposed to later agents. That work produced a negative full-task result under the memory conditions tested. The two studies address related questions, but neither experimentally proves the other.

5. Limitations and next steps

There are important boundaries on what this experiment can support:

• The benchmark is handwritten digit recognition, not an LLM or coding-agent environment. We do not claim gains for agent autonomy, reasoning or software engineering.
• Three seeds and one set of hyperparameters are not enough to establish an optimal continual-learning policy.
• Class-incremental learning can be particularly unforgiving to an online linear classifier; task design matters.
• Nearest-neighbor memory and SGD training are not compute-matched at inference time. High retrieval accuracy does not settle the systems tradeoff.

The next credible test would use authentic sequential agent tasks or recorded interaction trajectories, with old-task retention, unseen transfer, regression rates and cost tracked under a fixed harness. Until that experiment has been executed, it remains a research direction, not a finding.

Reproducibility. See the project folder for run.py, the run protocol and environment information, and results_summary.json for the reported measurements. The ReplayGate dashboard lets you compare all five stages interactively. It uses the same recorded values.