5 papers
DAR: Deontic Reasoning with Agentic Harnesses
Guangyao Dou, William Jurayj, Nils Holzenberger +1
Deontic reasoning is the task of answering questions by applying explicit rules and policies to case-specific facts, for example computing tax liability under a statute or determin…
DeonticBench: A Benchmark for Reasoning over Rules
Guangyao Dou, Luis Brena, Akhil Deo +4
Reasoning with complex, context-specific rules remains challenging for large language models (LLMs). In legal and policy settings, this manifests as deontic reasoning: reasoning ab…
Language Models and Logic Programs for Trustworthy Tax Reasoning
William Jurayj, Nils Holzenberger, Benjamin Van Durme
According to the United States Internal Revenue Service, ``the average American spends and 13 hours filing their taxes''. Even beyond the U.S., tax filing requires complex…
Can LLMs Identify Tax Abuse?
Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
We investigate whether large language models can discover and analyze U.S. tax-minimization strategies. This real-world domain challenges even seasoned human experts, and progress…
BLT: Can Large Language Models Handle Basic Legal Text?
Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
We find that the best publicly available LLMs like GPT-4 and Claude currently perform poorly on basic legal text handling. This motivates the creation of a benchmark consisting of…