Beyond Literal Token Overlap: Token Alignability for Multilinguality

Katharina Hämmerl, Tomasz Limisiewicz, Jindřich Libovický +1

Previous work has considered token overlap, or even similarity of token distributions, as predictors for multilinguality and cross-lingual knowledge transfer in language models. Ho…