5 papers
On Finding Inconsistencies in Documents
Charles J. Lovering, Seth Ebner, Brandon Smock +5
Professionals in academia, law, and finance audit their documents because inconsistencies can result in monetary, reputational, and scientific costs. Language models (LMs) have the…
Entropy-Driven Pre-Tokenization for Byte-Pair Encoding
Yifan Hu, Frank Liang, Dachuan Zhao +4
Byte-Pair Encoding (BPE) has become a widely adopted subword tokenization method in modern language models due to its simplicity and strong empirical performance across downstream…
BLEUBERI: BLEU is a surprisingly effective reward for instruction following
Yapei Chang, Yekyung Kim, Michael Krumdick +4
Reward models are central to aligning LLMs with human preferences, but they are costly to train, requiring large-scale human-labeled preference data and powerful pretrained LLM bac…
Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
Craig W. Schmidt, Varshini Reddy, Chris Tanner +1
Pre-tokenization, the initial step in many modern tokenization pipelines, segments text into smaller units called pretokens, typically splitting on whitespace and punctuation. Whil…
How Much is Enough? The Diminishing Returns of Tokenization Training Data
Varshini Reddy, Craig W. Schmidt, Yuval Pinter +1
Tokenization, a crucial initial step in natural language processing, is governed by several key parameters, such as the tokenization algorithm, vocabulary size, pre-tokenization st…