Showing 2025Show all
2 papers · 1 filter
cs.CL2025
Not a nuisance but a useful heuristic: Outlier dimensions favor frequent tokens in language models
Iuri Macocco, Nora Graichen, Gemma Boleda +1
We study last-layer outlier dimensions, i.e. dimensions that display extreme activations for the majority of inputs. We show that outlier dimensions arise in many different modern…
cs.CL2025
Prediction hubs are context-informed frequent tokens in LLMs
Beatrix M. G. Nielsen, Iuri Macocco, Marco Baroni
Hubness, the tendency for a few points to be among the nearest neighbours of a disproportionate number of other points, commonly arises when applying standard distance measures to…