MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling

Nathan Godey, Roman Castagné, Éric de la Clergerie +1

Static subword tokenization algorithms have been an essential component of recent works on language modeling. However, their static nature results in important flaws that degrade t…