A False Average: Pooled CoT-Monitor Accuracy Conceals a Reasoning-Dependent Fragility

Shikhar Shiromani, Leo Richter, Noah Siegel

Chain-of-thought (CoT) monitors are reported by their aggregate accuracy on a pool of reward hacks. We show that this number is a false average. On Terminal Wrench, about 77% of ha…