12 citations · 14 across the 12 of their papers we have counts for
12 papers
Joint Optimization for Greedy Longest-match Tokenization
Adhiraj Singh, Deepanshu Mody, Ghina Al Shdaifat +4
Recent work has shown that subword vocabularies can be trained to optimize compression for a specific inference rule rather than relying on greedy heuristics such as Byte Pair Enco…
Tokenization with Split Trees
Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4
We introduce Tokenization with Split Trees (ToaST), a subword tokenization method that directly optimizes compression under a new recursive inference procedure. ToaST greedily spli…
Tokenisation via Convex Relaxations
Jan Tempus, Philip Whittington, Craig W. Schmidt +2
Tokenisation is an integral part of the current NLP pipeline. Current tokenisation algorithms such as BPE and Unigram are greedy algorithms -- they make locally optimal decisions w…
Faster Superword Tokenization
Craig W. Schmidt, Chris Tanner, Yuval Pinter
Byte Pair Encoding (BPE) is a widely used tokenization algorithm, whose tokens cannot extend across pre-tokenization boundaries, functionally limiting it to representing at most fu…
The Effect of Scripts and Formats on LLM Numeracy
Varshini Reddy, Craig W. Schmidt, Seth Ebner +3
Large language models (LLMs) have achieved impressive proficiency in basic arithmetic, rivaling human-level performance on standard numerical tasks. However, little attention has b…
Entropy-Driven Pre-Tokenization for Byte-Pair Encoding
Yifan Hu, Frank Liang, Dachuan Zhao +4
Byte-Pair Encoding (BPE) has become a widely adopted subword tokenization method in modern language models due to its simplicity and strong empirical performance across downstream…
Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
Craig W. Schmidt, Varshini Reddy, Chris Tanner +1
Pre-tokenization, the initial step in many modern tokenization pipelines, segments text into smaller units called pretokens, typically splitting on whitespace and punctuation. Whil…
How Much is Enough? The Diminishing Returns of Tokenization Training Data
Varshini Reddy, Craig W. Schmidt, Yuval Pinter +1
Tokenization, a crucial initial step in natural language processing, is governed by several key parameters, such as the tokenization algorithm, vocabulary size, pre-tokenization st…
SEC-QA: A Systematic Evaluation Corpus for Financial QA
Viet Dac Lai, Michael Krumdick, Charles Lovering +3
The financial domain frequently deals with large numbers of long documents that are essential for daily operations. Significant effort is put towards automating financial data anal…
Greed is All You Need: An Evaluation of Tokenizer Inference Methods
Omri Uzan, Craig W. Schmidt, Chris Tanner +1
While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabul…
Tokenization Is More Than Compression
Craig W. Schmidt, Varshini Reddy, Haoran Zhang +4
Tokenization is a foundational step in natural language processing (NLP) tasks, bridging raw text and language models. Existing tokenization approaches like Byte-Pair Encoding (BPE…
Improving a tf-idf weighted document vector embedding
Craig W. Schmidt
We examine a number of methods to compute a dense vector embedding for a document in a corpus, given a set of word vectors such as those from word2vec or GloVe. We describe two met…