7 citations · 8 across the 12 of their papers we have counts for
14 papers
Tokenization with Split Trees
Craig W. Schmidt, Michael Krumdick, Adam Wiemerslage +4
We introduce Tokenization with Split Trees (ToaST), a subword tokenization method that directly optimizes compression under a new recursive inference procedure. ToaST greedily spli…
FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks
Michael Krumdick, Varshini Reddy, Shivani Chaudhary +8
As concerns surrounding AI-driven labor displacement intensify in knowledge-intensive sectors, existing benchmarks fail to measure performance on tasks that define practical profes…
Cost-Efficient Estimation of General Abilities Across Benchmarks
Michael Krumdick, Adam Wiemerslage, Seth Ebner +2
Thousands of diverse benchmarks have been developed to measure the quality of large language models (LLMs). Yet prior work has demonstrated that LLM performance is often sufficient…
On Finding Inconsistencies in Documents
Charles J. Lovering, Seth Ebner, Brandon Smock +5
Professionals in academia, law, and finance audit their documents because inconsistencies can result in monetary, reputational, and scientific costs. Language models (LMs) have the…
Complexity Scaling Laws for Neural Models using Combinatorial Optimization
Lowell Weissman, Michael Krumdick, A. Lynn Abbott
Recent work on neural scaling laws demonstrates that model performance scales predictably with compute budget, model size, and dataset size. In this work, we develop scaling laws b…
BLEUBERI: BLEU is a surprisingly effective reward for instruction following
Yapei Chang, Yekyung Kim, Michael Krumdick +4
Reward models are central to aligning LLMs with human preferences, but they are costly to train, requiring large-scale human-labeled preference data and powerful pretrained LLM bac…
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Michael Krumdick, Charles Lovering, Varshini Reddy +2
Reliable evaluation of large language models (LLMs) is critical as their deployment rapidly expands, particularly in high-stakes domains such as business and finance. The LLM-as-a-…
Language Model Probabilities are Not Calibrated in Numeric Contexts
Charles Lovering, Michael Krumdick, Viet Dac Lai +5
Some statements have one well-defined continuation (e.g., "the Eiffel Tower is in [Paris]"), whereas others have a natural distribution over multiple options (e.g., "the weighted c…
An Analysis of Multilingual FActScore
Kim Trong Vu, Michael Krumdick, Varshini Reddy +2
FActScore has gained popularity as a metric to estimate the factuality of long-form texts generated by Large Language Models (LLMs) in English. However, there has not been any work…
SEC-QA: A Systematic Evaluation Corpus for Financial QA
Viet Dac Lai, Michael Krumdick, Charles Lovering +3
The financial domain frequently deals with large numbers of long documents that are essential for daily operations. Significant effort is put towards automating financial data anal…
DocFinQA: A Long-Context Financial Reasoning Dataset
Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai +3
For large language models (LLMs) to be effective in the financial domain -- where each decision can have a significant impact -- it is necessary to investigate realistic tasks and…
BizBench: A Quantitative Reasoning Benchmark for Business and Finance
Rik Koncel-Kedziorski, Michael Krumdick, Viet Lai +3
Answering questions within business and finance requires reasoning, precision, and a wide-breadth of technical knowledge. Together, these requirements make this domain difficult fo…
A Graphical Approach to Document Layout Analysis
Jilin Wang, Michael Krumdick, Baojia Tong +5
Document layout analysis (DLA) is the task of detecting the distinct, semantic content within a document and correctly classifying these items into an appropriate category (e.g., t…
APRICOT: A Dataset of Physical Adversarial Attacks on Object Detection
Anneliese Braunegg, Amartya Chakraborty, Michael Krumdick +6
Physical adversarial attacks threaten to fool object detection systems, but reproducible research on the real-world effectiveness of physical patches and how to defend against them…