How Much Does Tokenization Affect Neural Machine Translation?

Miguel Domingo, Mercedes Garcıa-Martınez, Alexandre Helle +2

Tokenization or segmentation is a wide concept that covers simple processes such as separating punctuation from words, or more sophisticated processes such as applying morphologica…