180 citations · 352 across the 20 of their papers we have counts for
21 papers
Target Policy Optimization
Jean Kaddour
In RL, given a prompt, we sample a group of completions from a model and score them. Two questions follow: which completions should gain probability mass, and how should the parame…
Agentic Uncertainty Reveals Agentic Overconfidence
Jean Kaddour, Srijan Patel, Gbètondji Dovonon +3
Can AI agents predict whether they will succeed at a task? We study agentic uncertainty by eliciting success probability estimates before, during, and after task execution. All res…
From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation
Seokhee Hong, Sunkyoung Kim, Guijin Son +3
The development of Large Language Models (LLMs) requires robust benchmarks that encompass not only academic domains but also industrial fields to effectively evaluate their applica…
REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
Zafir Stojanovski, Oliver Stanley, Joe Sharratt +4
We introduce Reasoning Gym (RG), a library of reasoning environments for reinforcement learning with verifiable rewards. It provides over 100 data generators and verifiers spanning…
Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models
Georgy Tyukin, Gbetondji J-S Dovonon, Jean Kaddour +1
The inference demand for LLMs has skyrocketed in recent months, and serving models with low latencies remains challenging due to the quadratic input length complexity of the attent…
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim +30
Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to…
Are We Done with MMLU?
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong +13
Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates nu…
Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models
Jean Kaddour, Qi Liu
The in-context learning ability of large language models (LLMs) enables them to generalize to novel downstream tasks with relatively few labeled examples. However, they require eno…
Challenges and Applications of Large Language Models
Jean Kaddour, Joshua Harris, Maximilian Mozes +3
Large Language Models (LLMs) went from non-existent to ubiquitous in the machine learning discourse within a few years. Due to the fast pace of the field, it is difficult to identi…
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models
Jean Kaddour, Oscar Key, Piotr Nawrot +2
The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designe…
Early Weight Averaging meets High Learning Rates for LLM Pre-training
Sunny Sanyal, Atula Neerkaje, Jean Kaddour +2
Training Large Language Models (LLMs) incurs significant cost; hence, any strategy that accelerates model convergence is helpful. In this paper, we investigate the ability of a sim…
TTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Models
Yuwei Yin, Jean Kaddour, Xiang Zhang +4
Data augmentation has been established as an efficacious approach to supplement useful information for low-resource datasets. Traditional augmentation techniques such as noise inje…
The MiniPile Challenge for Data-Efficient Language Models
Jean Kaddour
The ever-growing diversity of pre-training text corpora has equipped language models with generalization capabilities across various downstream tasks. However, such diverse dataset…
Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases
Aengus Lynch, Gbètondji J-S Dovonon, Jean Kaddour +1
The problem of spurious correlations (SCs) arises when a classifier relies on non-predictive features that happen to be correlated with the labels in the training data. For example…
DAG Learning on the Permutahedron
Valentina Zantedeschi, Luca Franceschi, Jean Kaddour +2
We propose a continuous optimization framework for discovering a latent directed acyclic graph (DAG) from observational data. Our approach optimizes over the polytope of permutatio…
Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging
Jean Kaddour
Training vision or language models on large datasets can take days, if not weeks. We show that averaging the weights of the k latest checkpoints, each collected at the end of an ep…
Causal Machine Learning: A Survey and Open Problems
Jean Kaddour, Aengus Lynch, Qi Liu +2
Causal Machine Learning (CausalML) is an umbrella term for machine learning methods that formalize the data-generation process as a structural causal model (SCM). This perspective…
Evaluating Self-Supervised Learning for Molecular Graph Embeddings
Hanchen Wang, Jean Kaddour, Shengchao Liu +3
Graph Self-Supervised Learning (GSSL) provides a robust pathway for acquiring embeddings without expert labelling, a capability that carries profound implications for molecular gra…
When Do Flat Minima Optimizers Work?
Jean Kaddour, Linqing Liu, Ricardo Silva +1
Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic…
Causal Effect Inference for Structured Treatments
Jean Kaddour, Yuchen Zhu, Qi Liu +2
We address the estimation of conditional average treatment effects (CATEs) for structured treatments (e.g., graphs, images, texts). Given a weak condition on the effect, we propose…
Probabilistic Active Meta-Learning
Jean Kaddour, Steindór Sæmundsson, Marc Peter Deisenroth
Data-efficient learning algorithms are essential in many practical applications where data collection is expensive, e.g., in robotics due to the wear and tear. To address this prob…