Local Byte Fusion for Neural Machine Translation

Makesh Narsimhan Sreedhar, Xiangpeng Wan, Yu Cheng +1

Subword tokenization schemes are the dominant technique used in current NLP models. However, such schemes can be rigid and tokenizers built on one corpus do not adapt well to other…