Before I Compared the Models, I Had to Decide What Not to Count

I built a lab to compare the models running in my agent fleet on per-turn behavior like tool-error rate and re-edits. The numbers were the easy part. The work was everything that would have made a naive comparison lie — pooling two roles that are not comparable, averaging across policy regimes that are different treatments, folding in 871 threads I could not attribute to any model, and reading a bootstrap interval of [0, 0] as precision when it was a tie collapse. This is the list of things I refused to count, and why.