8 papers
A Human-Centric Evaluation of a Retrieval-Augmented Generation System for Explaining Quebec Insurance Contracts
David Beauchemin, Richard Khoury
With the rise of online insurance sales, consumers face a significant \enquote{advice gap}, requiring them to navigate complex legal contracts without expert guidance. This paper p…
Idiom Understanding as a Tool to Measure the Dialect Gap
David Beauchemin, Yan Tremblay, Mohamed Amine Youssef +1
The tasks of idiom understanding and dialect understanding are both well-established benchmarks in natural language processing. In this paper, we propose combining them, and using…
Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation
David Beauchemin, Richard Khoury
The digitization of insurance distribution in the Canadian province of Quebec, accelerated by legislative changes such as Bill 141, has created a significant "advice gap", leaving…
QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs
David Beauchemin, Pier-Luc Veilleux, Johanna-Pascale Roy +1
In this paper, we introduce the Quebec-French Benchmark of Linguistic Minimal Pairs (QFrBLiMP), a corpus designed to evaluate LLMs' linguistic knowledge of prominent grammatical ph…
COLE: a Comprehensive Benchmark for French Language Understanding Evaluation
David Beauchemin, Yan Tremblay, Mohamed Amine Youssef +1
To address the need for a more comprehensive evaluation of French Natural Language Understanding (NLU), we introduce COLE, a new benchmark composed of 23 diverse task covering a br…
JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences
David Beauchemin, Michelle Albert-Rochette, Richard Khoury +1
Simplifying text while preserving its meaning is a complex yet essential task, especially in sensitive domain applications like legal texts. When applied to a specialized field, li…