7 papers · 1 filter
Idiom Understanding as a Tool to Measure the Dialect Gap
David Beauchemin, Yan Tremblay, Mohamed Amine Youssef +1
The tasks of idiom understanding and dialect understanding are both well-established benchmarks in natural language processing. In this paper, we propose combining them, and using…
Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation
David Beauchemin, Richard Khoury
The digitization of insurance distribution in the Canadian province of Quebec, accelerated by legislative changes such as Bill 141, has created a significant "advice gap", leaving…
QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs
David Beauchemin, Pier-Luc Veilleux, Johanna-Pascale Roy +1
In this paper, we introduce the Quebec-French Benchmark of Linguistic Minimal Pairs (QFrBLiMP), a corpus designed to evaluate LLMs' linguistic knowledge of prominent grammatical ph…
COLE: a Comprehensive Benchmark for French Language Understanding Evaluation
David Beauchemin, Yan Tremblay, Mohamed Amine Youssef +1
To address the need for a more comprehensive evaluation of French Natural Language Understanding (NLU), we introduce COLE, a new benchmark composed of 23 diverse task covering a br…
JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences
David Beauchemin, Michelle Albert-Rochette, Richard Khoury +1
Simplifying text while preserving its meaning is a complex yet essential task, especially in sensitive domain applications like legal texts. When applied to a specialized field, li…
QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments
David Beauchemin, Richard Khoury
Large and Transformer-based language models perform outstandingly in various downstream tasks. However, there is limited understanding regarding how these models internalize lingui…