Dynamic Multi-Byte Prediction With Hierarchical Language Models

Abraham Toluwase Owodunni, Chibuzor Okocha, Christan Grant +2

Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byt…