From Smør-re-brød to Subwords: Training LLMs on Danish, One Morpheme at a Time

Mikkel Wildner Kildeberg, Emil Allerslev Schledermann, Nicolaj Larsen +1

The best performing transformer-based language models use subword tokenization techniques, such as Byte-Pair-Encoding (BPE). However, these approaches often overlook linguistic pri…