Batching BPE Tokenization Merges

Alexander P. Morgan

The Byte Pair Encoding algorithm can be safely batched to merge hundreds of pairs of tokens at a time when building up a tokenizer's vocabulary. This technique combined with reduci…