Vocabulary embeddings organize linguistic structure early in language model training

Isabel Papadimitriou, Jacob Prince

Large language models (LLMs) work by manipulating the geometry of input embedding vectors over multiple layers. Here, we ask: how are the input vocabulary representations of langua…