180 citations · 352 across the 20 of their papers we have counts for
8 papers · 1 filter
Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models
Jean Kaddour, Qi Liu
The in-context learning ability of large language models (LLMs) enables them to generalize to novel downstream tasks with relatively few labeled examples. However, they require eno…
Challenges and Applications of Large Language Models
Jean Kaddour, Joshua Harris, Maximilian Mozes +3
Large Language Models (LLMs) went from non-existent to ubiquitous in the machine learning discourse within a few years. Due to the fast pace of the field, it is difficult to identi…
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models
Jean Kaddour, Oscar Key, Piotr Nawrot +2
The computation necessary for training Transformer-based language models has skyrocketed in recent years. This trend has motivated research on efficient training algorithms designe…
Early Weight Averaging meets High Learning Rates for LLM Pre-training
Sunny Sanyal, Atula Neerkaje, Jean Kaddour +2
Training Large Language Models (LLMs) incurs significant cost; hence, any strategy that accelerates model convergence is helpful. In this paper, we investigate the ability of a sim…
TTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Models
Yuwei Yin, Jean Kaddour, Xiang Zhang +4
Data augmentation has been established as an efficacious approach to supplement useful information for low-resource datasets. Traditional augmentation techniques such as noise inje…
The MiniPile Challenge for Data-Efficient Language Models
Jean Kaddour
The ever-growing diversity of pre-training text corpora has equipped language models with generalization capabilities across various downstream tasks. However, such diverse dataset…
Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases
Aengus Lynch, Gbètondji J-S Dovonon, Jean Kaddour +1
The problem of spurious correlations (SCs) arises when a classifier relies on non-predictive features that happen to be correlated with the labels in the training data. For example…
DAG Learning on the Permutahedron
Valentina Zantedeschi, Luca Franceschi, Jean Kaddour +2
We propose a continuous optimization framework for discovering a latent directed acyclic graph (DAG) from observational data. Our approach optimizes over the polytope of permutatio…