How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?

Ali Araabi, Christof Monz, Vlad Niculae

Neural Machine Translation (NMT) is an open vocabulary problem. As a result, dealing with the words not occurring during training (a.k.a. out-of-vocabulary (OOV) words) have long b…