12 citations · 14 across the 12 of their papers we have counts for
5 papers · 1 filter
Joint Optimization for Greedy Longest-match Tokenization
Adhiraj Singh, Deepanshu Mody, Ghina Al Shdaifat +4
Recent work has shown that subword vocabularies can be trained to optimize compression for a specific inference rule rather than relying on greedy heuristics such as Byte Pair Enco…
Tokenization with Split Trees
Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4
We introduce Tokenization with Split Trees (ToaST), a subword tokenization method that directly optimizes compression under a new recursive inference procedure. ToaST greedily spli…
Tokenisation via Convex Relaxations
Jan Tempus, Philip Whittington, Craig W. Schmidt +2
Tokenisation is an integral part of the current NLP pipeline. Current tokenisation algorithms such as BPE and Unigram are greedy algorithms -- they make locally optimal decisions w…
Faster Superword Tokenization
Craig W. Schmidt, Chris Tanner, Yuval Pinter
Byte Pair Encoding (BPE) is a widely used tokenization algorithm, whose tokens cannot extend across pre-tokenization boundaries, functionally limiting it to representing at most fu…
The Effect of Scripts and Formats on LLM Numeracy
Varshini Reddy, Craig W. Schmidt, Seth Ebner +3
Large language models (LLMs) have achieved impressive proficiency in basic arithmetic, rivaling human-level performance on standard numerical tasks. However, little attention has b…