10 papers
UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval
Tobias Schimanski, Stefanie Lewandowski, Christian Woerle +3
Conventional information retrieval is concerned with identifying the relevance of texts for a given query. Yet, the conventional definition of relevance is dominated by aspects of…
ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models
Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu +8
LLMs can solve complex tasks by generating long, multi-step reasoning chains. Test-time scaling (TTS) can further improve performance by sampling multiple variants of intermediate…
LEXam: Benchmarking Legal Reasoning on 340 Law Exams
Yu Fan, Jingwei Ni, Jakob Merane +14
Long-form legal reasoning remains a key challenge for large language models (LLMs) in spite of recent advances in test-time scaling. To address this, we introduce LEXam, a novel be…
Actors, Frames and Arguments: A Multi-Decade Computational Analysis of Climate Discourse in Financial News using Large Language Models
Ruiran Su, Janet B. Pierrehumbert, Markus Leippold
Financial news media shapes trillion-dollar climate investment decisions, yet discourse in this elite domain remains underexplored. We analyze two decades of climate-related articl…
Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?
Jingwei Ni, Yu Fan, Vilém Zouhar +6
Variation in human annotation (i.e., disagreements) is common in NLP, often reflecting important information like task subjectivity and sample ambiguity. Modeling this variation is…
pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs
Tobias Schimanski, Imene Kolli, Yu Fan +4
PDFs are the second-most used document type on the internet (after HTML). Yet, existing QA datasets commonly start from text sources or only address specific domains. In this paper…