Understanding and Mitigating Tokenization Bias in Language Models

Buu Phan, Marton Havasi, Matthew Muckley +1

State-of-the-art language models are autoregressive and operate on subword units known as tokens. Specifically, one must encode the conditioning string into a list of tokens before…