41 citations · 59 across the 9 of their papers we have counts for
9 papers
Local and Global Regimes of Geometric Complexity in Language Model Representations
Arwa Osman, Marco Baroni, Iuri Macocco
Intrinsic dimensionality (ID) is widely used to probe the representational complexity of language models, but it remains unclear whether ID differences reflect properties of langua…
On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study
Iuri Macocco, Pau Rodríguez, Arno Blaas +3
Controlling the output of Large Language Models (LLMs) is a central challenge for their reliable deployment, yet a clear understanding of the involved trade-offs remains elusive. C…
Tracing Computation Density in LLMs
Corentin Kervadec, Iuliia Lysova, Iuri Macocco +2
Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs, but it is not clear that they exploit their f…
Not a nuisance but a useful heuristic: Outlier dimensions favor frequent tokens in language models
Iuri Macocco, Nora Graichen, Gemma Boleda +1
We study last-layer outlier dimensions, i.e. dimensions that display extreme activations for the majority of inputs. We show that outlier dimensions arise in many different modern…
Prediction hubs are context-informed frequent tokens in LLMs
Beatrix M. G. Nielsen, Iuri Macocco, Marco Baroni
Hubness, the tendency for a few points to be among the nearest neighbours of a disproportionate number of other points, commonly arises when applying standard distance measures to…
Emergence of a High-Dimensional Abstraction Phase in Language Transformers
Emily Cheng, Diego Doimo, Corentin Kervadec +4
A language model (LM) is a mapping from a linguistic context to an output token. However, much remains to be known about this mapping, including how its geometric properties relate…
Scale adaptive and robust intrinsic dimension estimation via optimal neighbourhood identification
Antonio Di Noia, Iuri Macocco, Aldo Glielmo +2
The Intrinsic Dimension (ID) is a key concept in unsupervised learning and feature selection, as it is a lower bound to the number of variables which are necessary to describe a sy…
Intrinsic dimension estimation for discrete metrics
Iuri Macocco, Aldo Glielmo, Jacopo Grilli +1
Real world-datasets characterized by discrete features are ubiquitous: from categorical surveys to clinical questionnaires, from unweighted networks to DNA sequences. Nevertheless,…
DADApy: Distance-based Analysis of DAta-manifolds in Python
Aldo Glielmo, Iuri Macocco, Diego Doimo +6
DADApy is a python software package for analysing and characterising high-dimensional data manifolds. It provides methods for estimating the intrinsic dimension and the probability…