~~You read that as ongoing metrics against a the last pre-patch run?~~

nvm the caveat that I missed is the fixed task.

optimism

As they use the harness, they feed the traces and output to what they call a 

 which proposes patches to the modifiable surface of the harness (the stuff that 

 lists), they then evaluate the results produced by each patch relative to the pre-patch harness using a "hidden selection set" of problems, accept or reject each patch based on whether it improves the harness performance or not, then at the end of that loop make sure it's not overfitting to the hidden selection set by testing it against a final "holdout set."

Evolving harnesses versus static ones

As they use the harness, they feed the traces and output to what they call a `proposer` which proposes patches to the modifiable surface of the harness (the stuff that @optimism lists), they then evaluate the results produced by each patch relative to the pre-patch harness using a "hidden selection set" of problems, accept or reject each patch based on whether it improves the harness performance or not, then at the end of that loop make sure it's not overfitting to the hidden selection set by testing it against a final "holdout set."

They have a cool picture of the process.

![](https://m.stacker.news/154567)