Open the reproducible Colab notebook · Source and methodology · Raw held-out results

The motivation: a moving evaluator

In DQN, a model learns from bootstrapped targets based on estimates from another model. If the estimating network and its target change rapidly together, training can oscillate. A target network, Double DQN, and the dueling value/advantage architecture address different aspects of that instability.

Recursive self-improvement (RSI) systems face a superficially similar issue: they propose modifications, use measurements to decide which to keep, and then propose further modifications from the changed system. I wanted to test the mechanism, not assume the analogy was already proven.

The question I put into code was:

If a controller learns to pick successive program optimizations, do RL target-network techniques improve its ability to choose good modifications on workloads it has not trained on?

That is narrower than creating a self-improving language model. It is a useful engineering question, however, because it forces an evaluator, candidate modifications, real execution, regression checks, and a held-out test.

The executable code gym

I implemented k-nearest-neighbor search with 24 configurations assembled from executable NumPy building blocks: naive broadcasting, matrix-multiplication distance computation, chunked matrix multiplication, full sorting versus partial selection, and precision/chunk-size choices. Not every configuration is a distinct source-level algorithm: chunk size does not affect the non-chunked variants.

At each step, the controller can propose one of six edits to the current program configuration. We execute each candidate family on benchmark workloads beforehand, check that its nearest-neighbor indices match a double-precision brute-force reference, and cache measured latencies. During RL training, action feedback comes from that table of real runs. A faster valid edit is accepted; a slower edit is rejected. Each episode allows five proposals.

The distinction matters: the RL rollout uses cached measurements, not live recompilation, and no LLM writes the source code. This isolates the edit-selection problem while keeping the reward attached to working code.

current kNN implementation
        │
        ▼
RL controller picks a code edit
        │
        ▼
candidate configuration → verified measured latency
        │
        ├─ faster and correct → accept
        └─ slower or incorrect → reject
        │
        ▼
log speed improvement → replay buffer → Q-network update
        └───────────────────────────────────────────┐
                         next edit ◀────────────────┘

What I ablated

Every policy gets a five-edit budget on the same initial code. I trained on four input shapes, froze network parameters, and measured outcomes on three different held-out shapes. Six seeded runs per method yielded 90 held-out method/workload/seed outcomes.

All learned methods use replay, the same network-width budget, 75 training episodes per seed, and the same action budget at test time. The combined arm is not a clean single-variable test of dueling or EMA, and should not be interpreted as one.

The connected Colab runtime was a Tesla T4. PyTorch used its GPU for network training; the benchmarked NumPy kernels ran on the runtime’s CPU. The six notebook code cells all executed, with 96/96 calibration and 72/72 held-out correctness checks passing.

The results

On a fresh Colab T4 rerun, online-only DQN ranked first, with a geometric 12.38× speedup versus 10.22× for random. Its seed-paired mean log-speedup difference was +0.191. A two-sided Wilcoxon comparison yielded p = 0.0625 before correction; with Holm correction for the four comparisons, all adjusted p-values were 0.25. In this rerun the combined dueling/Double/EMA model ranked below each other learned controller. This variability itself is a reason not to overinterpret any single run.

My conclusion is not that soft updates work for RSI. The rerun found no statistically reliable improvement from target networks, and its ranking reversed the earlier run. It cannot disentangle which of the combined mechanisms matters, and six seeds are too few for confidence. The large absolute speedups are unsurprising because the original program intentionally uses a costly broadcast distance computation; optimizing that baseline does not establish an advance over an optimized nearest-neighbor library.

The most important failure modes

Cached metrics create a simpler world than an autonomous agent faces. In real automated AI research, a proposed change might require training, cause nondeterministic failures, or exploit the evaluator. Here the entire finite family can be benchmarked in advance. That makes this a test of RL selection, not automated hypothesis discovery.

Correctness is necessary but not sufficient. The oracle checks exact neighbor indices on sampled inputs. Production-ready optimizations would need much broader correctness and numerical-stability tests, different hardware, resource limits, and careful measurement methodology.

Held-out workload shapes are not held-out code families. Every held-out evaluation still chooses among the same 24 implementation configurations. Transfer to new algorithm families remains unknown.

The comparison combines mechanisms. Dueling Double DQN + EMA outperformed the random baseline descriptively. That does not identify whether dueling, Double DQN, EMA, replay, or random variation drove the outcome. A factorial ablation with many more seeds would be needed.

Connection to the larger research problem

Darwin Gödel Machine uses empirical feedback to evolve coding agents. RRSI studies regularized evolution of an agent’s harness and shows why improvement on the evolution tasks can fail to transfer. Eureka uses LLM-authored code in an iterative reward-design loop. These are much richer systems than this notebook.

The next meaningful test would let an LLM propose genuinely new code patches, run sandboxed tests and performance measurements, and train a conservative controller to choose which hypotheses to spend the next compute budget on. At that point we could compare a frozen critic, EMA teacher, Double DQN, and acceptance gates while measuring both performance and harmful regressions. Such a study would need leakage-resistant held-out tasks and budget-matched search.

For now, the outcome is a working and auditable RL-governed code-optimization prototype. The most transferable lesson is methodological: verify that improvements actually run, separate calibration from held-out evaluation, publish negative results, and avoid inferring a new algorithm from one appealing chart.

Notebook · CSV data and exact statistical report · Methodology and reproduction

A model can learn a new task and, in the process, lose something it previously knew. We tested this under a simple, controlled sequence of five classification tasks built from 1,797 real handwritten digit images. An online classifier trained only on the newest task retained just 19.0% accuracy across all previously seen digit classes at the end. Keeping a small memory of earlier examples changed the outcome: rehearsal achieved 91.3%, and a retrieval-only memory achieved 93.0%. These are measured averages across three seeded runs, not illustrative values.

The result is useful as a concrete illustration of catastrophic forgetting. It is not evidence that the same approaches improve LLMs or coding agents. The purpose here was to examine the retention–adaptation tradeoff before attempting an agent-scale evaluation.

1. The question

Suppose a system works well today, encounters a new task tomorrow, and changes how it behaves. That change might live in its model weights, in a replay buffer used during training, or in an external memory consulted at inference time. All three can help it adapt, but they preserve old abilities in different ways.

The difficulty is that performance on the latest task is not a reliable measure of continual learning. A system might score highly on its new task while failing at an old one. To see that failure rather than hide it, we need to evaluate earlier tasks again after each new training stage.

For this first experiment, we deliberately use a small real-world classification setting instead of an expensive agent benchmark. This isolates the question of retention; it does not pretend that classifying digits captures every complication of learning from agent trajectories.

2. Experimental setup

2.1 Dataset and task sequence

The data are the UCI Optical Recognition of Handwritten Digits images, loaded through sklearn.datasets.load_digits(). There are 1,797 authentic handwritten examples, each represented by 64 pixel intensities. No synthetic or handwritten-for-this-demo evaluation cases were added.

Each seed has a stratified 75% training / 25% held-out testing split. The learner sees five tasks in this fixed order:

2.2 Three ways to handle past experience

Weights only. A multiclass logistic classifier receives 120 stochastic-gradient minibatch updates per stage, using only examples of the two most recent classes. It has no way to revisit earlier examples. This is our forgetting-prone baseline.

Reservoir replay. The classifier gets the same 120 parameter-update steps per stage, but can fill half a minibatch from an online reservoir of at most 160 previously encountered training examples. The reservoir samples uniformly from experience seen so far. Learning still happens in model weights, while memory supplies rehearsal data.

Exemplar memory. A distance-weighted, 3-nearest-neighbor classifier predicts labels using a reservoir of 160 real training examples. It does not use the same parameter-update process. This makes it a useful retrieval-only reference, but not an equal-compute comparison to the other models.

We used seeds 7, 13, and 29 and evaluated every method on that seed’s untouched test split. The full Python implementation specifies the data, sampling, training loop, and evaluation procedure.

3. Findings

3.1 Learning the new task is not retaining the old ones

At the final stage, the weight-only model is still good at classifying the new digits, but its accuracy on the first task has dropped to 0%. This is the central failure the experiment was meant to expose: a high latest-task score can coexist with almost complete loss of earlier ability.

3.2 Memory changes the retention–adaptation balance

The replay model preserves much more of the first task while learning the final task reasonably well. The retrieval-only system keeps the earliest task almost intact, but scores lower on the newest one. These numbers should not be used to declare a universal winner: the methods have different computational costs and inductive biases.

4. Interpretation

A growing system has at least two separate obligations: acquire new behavior and avoid unnecessary regressions. The baseline optimizes the first with almost no protection for the second. Replay reintroduces earlier data during adaptation; exemplar retrieval avoids overwriting a learned classifier altogether.

This is a useful reminder for work on continual learning and agent memory, where it is tempting to measure only performance on the latest benchmark. A more complete evaluation should ask what happens to older tasks, whether retained information transfers to unseen situations, and what it costs in compute, storage, and inference time.

My separate coding-agent memory study examines another piece of this problem: whether earlier experiences should be promoted and exposed to later agents. That work produced a negative full-task result under the memory conditions tested. The two studies address related questions, but neither experimentally proves the other.

5. Limitations and next steps

There are important boundaries on what this experiment can support:

• The benchmark is handwritten digit recognition, not an LLM or coding-agent environment. We do not claim gains for agent autonomy, reasoning or software engineering.
• Three seeds and one set of hyperparameters are not enough to establish an optimal continual-learning policy.
• Class-incremental learning can be particularly unforgiving to an online linear classifier; task design matters.
• Nearest-neighbor memory and SGD training are not compute-matched at inference time. High retrieval accuracy does not settle the systems tradeoff.

The next credible test would use authentic sequential agent tasks or recorded interaction trajectories, with old-task retention, unseen transfer, regression rates and cost tracked under a fixed harness. Until that experiment has been executed, it remains a research direction, not a finding.

Reproducibility. See the project folder for run.py, the run protocol and environment information, and results_summary.json for the reported measurements. The ReplayGate dashboard lets you compare all five stages interactively. It uses the same recorded values.

Try the deployed promotion-policy demo · Get the runnable Python code

The problem

An agent makes mistakes, studies its training failures, and proposes a new rule for future tasks. It gets more of those training examples right. Does that mean the rule deserves to become part of the agent’s permanent playbook?

Not necessarily. A rule can explain visible failures while overfitting them. It might also break some tasks the original agent had already solved. I built SkillFalsifier to ask what evidence is needed before a self-generated skill is promoted.

The testbed

The GitHub project is a runnable Python CLI harness. An agent classifies hackathon-rules scenarios as allowed, banned, or needs review. Its predictions are scored by exact-match rules—not a free-form LLM judge.

It works with three data splits:

A real LLM proposes candidate skills from training failures. Each candidate is evaluated on the full set. Scores and promotion decisions are recorded in the CLI reports.

Naive improvement versus falsification

The baseline naive promotion policy accepts a skill whenever it improves training accuracy.

The stricter SkillFalsifier gate requires three conditions: training accuracy must not decrease, accuracy on held-out examples must improve, and no previously correct adversarial case may become wrong.

That distinction is the project’s central failure mode: a skill can increase the score on the examples that inspired it while reducing accuracy elsewhere. Training gains alone are not evidence of reliable improvement.

The interactive demo illustrates the exact deterministic promotion logic in the source. The demo inputs are user-controlled hypotheticals, not fabricated experiment results. To reproduce empirical results, run the repository on its actual datasets and model backends.

Limits and why I care

This project is a small controlled testbed, not a universal theorem about agent self-improvement. The particularly conservative no-adversarial-regressions criterion may discard genuinely useful candidate skills. Larger datasets, multiple models, and repeated trials would be needed to evaluate production reliability.

What transfers is the evaluation question: if an agent learns a reusable skill, memory, or procedure from its own experience, how do we check that the update actually helps on work it hasn’t seen?

A related research project on which coding-agent experiences deserve to persist is described in my agent memory research note.

Code · Deployed decision demo

Headings are cool

You can have many headings

Aren’t headings cool?

Headings are cool

You can have many headings

Aren’t headings cool?

Headings are cool

You can have many headings

Aren’t headings cool?

Headings are cool

You can have many headings

Aren’t headings cool?