Token embeddings violate the manifold hypothesis

Michael Robinson, Sourya Dey, Tony Chiang

A full understanding of the behavior of a large language model (LLM) requires our grasp of its input token space. If this space differs from our assumptions, our comprehension of a…