All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling

Emanuele Marconato, Sébastien Lachapelle, Sebastian Weichwald +1

We analyze identifiability as a possible explanation for the ubiquity of linear properties across language models, such as the vector difference between the representations of "eas…