13 papers
The Effect of Scripts and Formats on LLM Numeracy
Varshini Reddy, Craig W. Schmidt, Seth Ebner +3
Large language models (LLMs) have achieved impressive proficiency in basic arithmetic, rivaling human-level performance on standard numerical tasks. However, little attention has b…
Tokenization with Split Trees
Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4
We introduce Tokenization with Split Trees (ToaST), a subword tokenization method that directly optimizes compression under a new recursive inference procedure. ToaST greedily spli…
FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks
Michael Krumdick, Varshini Reddy, Shivani Chaudhary +8
As concerns surrounding AI-driven labor displacement intensify in knowledge-intensive sectors, existing benchmarks fail to measure performance on tasks that define practical profes…
Faster Superword Tokenization
Craig W. Schmidt, Chris Tanner, Yuval Pinter
Byte Pair Encoding (BPE) is a widely used tokenization algorithm, whose tokens cannot extend across pre-tokenization boundaries, functionally limiting it to representing at most fu…
Cost-Efficient Estimation of General Abilities Across Benchmarks
Michael Krumdick, Adam Wiemerslage, Seth Ebner +2
Thousands of diverse benchmarks have been developed to measure the quality of large language models (LLMs). Yet prior work has demonstrated that LLM performance is often sufficient…
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Michael Krumdick, Charles Lovering, Varshini Reddy +2
Reliable evaluation of large language models (LLMs) is critical as their deployment rapidly expands, particularly in high-stakes domains such as business and finance. The LLM-as-a-…