8 citations · 19 across the 10 of their papers we have counts for
10 papers
Mining Legal Arguments in U.S. Corporate Case Law
Luis Brena, William Jurayj, Gregory Deyesu +3
Legal argument mining supports passage classification, retrieval, and argument completion. This work introduces an expert-annotated dataset of 42 U.S. federal tax opinions on corpo…
Can LLMs Identify Tax Abuse?
Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
We investigate whether large language models can discover and analyze U.S. tax-minimization strategies. This real-world domain challenges even seasoned human experts, and progress…
LLMs Provide Unstable Answers to Legal Questions
Andrew Blair-Stanek, Benjamin Van Durme
An LLM is stable if it reaches the same conclusion when asked the identical question multiple times. We find leading LLMs like gpt-4o, claude-3.5, and gemini-1.5 are unstable when…
Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluations
Abe Bohan Hou, William Jurayj, Nils Holzenberger +2
Large Language Models (LLMs) show promise as a writing aid for professionals performing legal analyses. However, LLMs can often hallucinate in this setting, in ways difficult to re…
CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generation
Abe Bohan Hou, Orion Weller, Guanghui Qin +5
Legal professionals need to write analyses that rely on citations to relevant precedents, i.e., previous case decisions. Intelligent systems assisting legal professionals in writin…
BLT: Can Large Language Models Handle Basic Legal Text?
Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
We find that the best publicly available LLMs like GPT-4 and Claude currently perform poorly on basic legal text handling. This motivates the creation of a benchmark consisting of…
OpenAI Cribbed Our Tax Example, But Can GPT-4 Really Do Tax?
Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
The authors explain where OpenAI got the tax law example in its livestream demonstration of GPT-4, why GPT-4 got the wrong answer, and how it fails to reliably calculate taxes.
InteractiveIE: Towards Assessing the Strength of Human-AI Collaboration in Improving the Performance of Information Extraction
Ishani Mondal, Michelle Yuan, Anandhavelu N +5
Learning template based information extraction from documents is a crucial yet difficult task. Prior template-based IE approaches assume foreknowledge of the domain templates; howe…
Can GPT-3 Perform Statutory Reasoning?
Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
Statutory reasoning is the task of reasoning with facts and statutes, which are rules written in natural language by a legislature. It is a basic legal skill. In this paper we expl…
A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering
Nils Holzenberger, Andrew Blair-Stanek, Benjamin Van Durme
Legislation can be viewed as a body of prescriptive rules expressed in natural language. The application of legislation to facts of a case we refer to as statutory reasoning, where…