Concepts Whisper: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations

Pratyush Acharya, Nuraj Rimal, Habish Dhakal

We find that transformer concept representations systematically anti-concentrate in the spectral tail of the unembedding covariance, encoding word-level concepts in low-variance di…