When a Zero-Parameter Cache Overtakes a Transformer

A count table over the current document has no parameters and no training. Somewhere between 60 and 250 tokens of document it passes a 1.43M-parameter transformer, and by 1000 tokens it wins top-1 by 0.064, a 43% relative margin. Adding the transformer on top then buys 0.002.