Measuring Intrinsic Dimension of Token Embeddings

Takuya Kataiwa, Cho Hakaze, Tetsushi Ohki

In this study, we measure the Intrinsic Dimension (ID) of token embedding to estimate the intrinsic dimensions of the manifolds spanned by the representations, so as to evaluate th…