Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
Causal Dimensionality of Transformer Representations: Measurement, Scaling, and Layer Structure
Nilesh Sarkar, Dawar Jyoti Deka
Sparse autoencoders (SAEs) decompose transformer residual streams into interpretable feature dictionaries, yet the relationship between SAE width and causal influence on model outp…
cs.LG2026
Geometric Limits of Knowledge Distillation: A Minimum-Width Theorem via Superposition Theory
Nilesh Sarkar, Dawar Jyoti Deka
Knowledge distillation compresses large teachers into smaller students, but performance saturates at a loss floor that persists across training methods and objectives. We argue thi…