Why bother with geometry? On the relevance of linear decompositions of Transformer embeddings

Timothee Mickus, Raúl Vázquez

A recent body of work has demonstrated that Transformer embeddings can be linearly decomposed into well-defined sums of factors, that can in turn be related to specific network inp…