Did the agent actually improve?
A self-generated skill can improve performance on its training examples and still fail on unseen tasks. Try the actual promotion logic from my SkillFalsifier source code. Inputs are illustrative examples, not empirical study results.
Adjust the evidence
Compared with the baseline on the cases the agent learned from.
Compared with the baseline on unseen cases.
Previously correct adversarial cases that became incorrect.
Naive policy
Promote if training improves (> 0).
SkillFalsifier gate
Training must not fall; held-out must improve; adversarial regressions must be zero.
About the real experiment
The public Python project runs an LLM-powered skill generator, compares baselines, evaluates candidates on training, held-out, and adversarial examples, and applies these two promotion rules. This webpage only reproduces the decision logic. It does not run a model, simulate experiment scores, or claim that the example numbers were measured.