LLBPE: Linked-List Based GPU-Parallel BPE Tokenizer

Aditya Kovilur, Varun Chandra Shekar, Ariful Azad

Every LLM inference begins with tokenization, which converts raw input bytes into the discrete token sequence the model consumes. For text, this step is often implemented using Byt…