2 citations · 3 across the 17 of their papers we have counts for
18 papers
Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment
Joshua Wong, Chris Tanner
Transformer-based models excel at Automatic Readability Assessment (ARA), yet feature-based models remain in active use because their predictions tie back to linguistic properties.…
Tokenization with Split Trees
Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4
We introduce Tokenization with Split Trees (ToaST), a subword tokenization method that directly optimizes compression under a new recursive inference procedure. ToaST greedily spli…
FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks
Michael Krumdick, Varshini Reddy, Shivani Chaudhary +8
As concerns surrounding AI-driven labor displacement intensify in knowledge-intensive sectors, existing benchmarks fail to measure performance on tasks that define practical profes…
Faster Superword Tokenization
Craig W. Schmidt, Chris Tanner, Yuval Pinter
Byte Pair Encoding (BPE) is a widely used tokenization algorithm, whose tokens cannot extend across pre-tokenization boundaries, functionally limiting it to representing at most fu…
Cost-Efficient Estimation of General Abilities Across Benchmarks
Michael Krumdick, Adam Wiemerslage, Seth Ebner +2
Thousands of diverse benchmarks have been developed to measure the quality of large language models (LLMs). Yet prior work has demonstrated that LLM performance is often sufficient…
The Effect of Scripts and Formats on LLM Numeracy
Varshini Reddy, Craig W. Schmidt, Seth Ebner +3
Large language models (LLMs) have achieved impressive proficiency in basic arithmetic, rivaling human-level performance on standard numerical tasks. However, little attention has b…
On Finding Inconsistencies in Documents
Charles J. Lovering, Seth Ebner, Brandon Smock +5
Professionals in academia, law, and finance audit their documents because inconsistencies can result in monetary, reputational, and scientific costs. Language models (LMs) have the…
Entropy-Driven Pre-Tokenization for Byte-Pair Encoding
Yifan Hu, Frank Liang, Dachuan Zhao +4
Byte-Pair Encoding (BPE) has become a widely adopted subword tokenization method in modern language models due to its simplicity and strong empirical performance across downstream…
BLEUBERI: BLEU is a surprisingly effective reward for instruction following
Yapei Chang, Yekyung Kim, Michael Krumdick +4
Reward models are central to aligning LLMs with human preferences, but they are costly to train, requiring large-scale human-labeled preference data and powerful pretrained LLM bac…
Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
Craig W. Schmidt, Varshini Reddy, Chris Tanner +1
Pre-tokenization, the initial step in many modern tokenization pipelines, segments text into smaller units called pretokens, typically splitting on whitespace and punctuation. Whil…
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Michael Krumdick, Charles Lovering, Varshini Reddy +2
Reliable evaluation of large language models (LLMs) is critical as their deployment rapidly expands, particularly in high-stakes domains such as business and finance. The LLM-as-a-…
How Much is Enough? The Diminishing Returns of Tokenization Training Data
Varshini Reddy, Craig W. Schmidt, Yuval Pinter +1
Tokenization, a crucial initial step in natural language processing, is governed by several key parameters, such as the tokenization algorithm, vocabulary size, pre-tokenization st…
Language Model Probabilities are Not Calibrated in Numeric Contexts
Charles Lovering, Michael Krumdick, Viet Dac Lai +5
Some statements have one well-defined continuation (e.g., "the Eiffel Tower is in [Paris]"), whereas others have a natural distribution over multiple options (e.g., "the weighted c…
SEC-QA: A Systematic Evaluation Corpus for Financial QA
Viet Dac Lai, Michael Krumdick, Charles Lovering +3
The financial domain frequently deals with large numbers of long documents that are essential for daily operations. Significant effort is put towards automating financial data anal…
Greed is All You Need: An Evaluation of Tokenizer Inference Methods
Omri Uzan, Craig W. Schmidt, Chris Tanner +1
While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabul…
Tokenization Is More Than Compression
Craig W. Schmidt, Varshini Reddy, Haoran Zhang +4
Tokenization is a foundational step in natural language processing (NLP) tasks, bridging raw text and language models. Existing tokenization approaches like Byte-Pair Encoding (BPE…
DocFinQA: A Long-Context Financial Reasoning Dataset
Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai +3
For large language models (LLMs) to be effective in the financial domain -- where each decision can have a significant impact -- it is necessary to investigate realistic tasks and…
BizBench: A Quantitative Reasoning Benchmark for Business and Finance
Rik Koncel-Kedziorski, Michael Krumdick, Viet Lai +3
Answering questions within business and finance requires reasoning, precision, and a wide-breadth of technical knowledge. Together, these requirements make this domain difficult fo…