BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition

Hyunsik Kim, Haeri Kim, Munhak Lee +1

Multilingual automatic speech recognition (ASR) requires tokenization that efficiently covers many writing systems. Byte-level BPE (BBPE) using UTF-8 is widely adopted for its lang…