11 citations · 20 across the 15 of their papers we have counts for
17 papers
Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM
Adam Zachary Wasserman, David Beauchemin
We submit MéTRON-FR, a 125M GPT-2 pretrained on 92.47M words of French, to the BabyLM 2026 Strict track. It scores 85.97 +/- 0.17% on QFrBLiMP (a native Quebec-French benchmark of…
A Human-Centric Evaluation of a Retrieval-Augmented Generation System for Explaining Quebec Insurance Contracts
David Beauchemin, Richard Khoury
With the rise of online insurance sales, consumers face a significant \enquote{advice gap}, requiring them to navigate complex legal contracts without expert guidance. This paper p…
Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation
David Beauchemin, Richard Khoury
The digitization of insurance distribution in the Canadian province of Quebec, accelerated by legislative changes such as Bill 141, has created a significant "advice gap", leaving…
COLE: a Comprehensive Benchmark for French Language Understanding Evaluation
David Beauchemin, Yan Tremblay, Mohamed Amine Youssef +1
To address the need for a more comprehensive evaluation of French Natural Language Understanding (NLU), we introduce COLE, a new benchmark composed of 23 diverse task covering a br…
Idiom Understanding as a Tool to Measure the Dialect Gap
David Beauchemin, Yan Tremblay, Mohamed Amine Youssef +1
The tasks of idiom understanding and dialect understanding are both well-established benchmarks in natural language processing. In this paper, we propose combining them, and using…
QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs
David Beauchemin, Pier-Luc Veilleux, Johanna-Pascale Roy +1
In this paper, we introduce the Quebec-French Benchmark of Linguistic Minimal Pairs (QFrBLiMP), a corpus designed to evaluate LLMs' linguistic knowledge of prominent grammatical ph…