3 papers
cs.CL2026
Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMs
Corentin Kervadec, Iuliia Lysova, Marco Baroni +1
Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs. Several studies on LLM efficiency optimizatio…
cs.CL2025
Not a nuisance but a useful heuristic: Outlier dimensions favor frequent tokens in language models
Iuri Macocco, Nora Graichen, Gemma Boleda +1
We study last-layer outlier dimensions, i.e. dimensions that display extreme activations for the majority of inputs. We show that outlier dimensions arise in many different modern…
cs.CL2024
Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM
Haw-Shiuan Chang, Nanyun Peng, Mohit Bansal +2
Contrastive decoding (CD) (Li et al., 2023) improves the next-token distribution of a large expert language model (LM) using a small amateur LM. Although CD is applied to various L…