The structure of the token space for large language models

Michael Robinson, Sourya Dey, Shauna Sweet

Large language models encode the correlational structure present in natural language by fitting segments of utterances (tokens) into a high dimensional ambient latent space upon wh…