Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling

Eric Egli, Matteo Manica, Jannis Born

Bytes form the basis of the digital world and thus are a promising building block for multimodal foundation models. Recently, Byte Language Models (BLMs) have emerged to overcome t…