794 citations · 5.7k across the 143 of their papers we have counts for
20 papers · 1 filter
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
Lili Yu, Bowen Shi, Ramakanth Pasunuru +24
We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented, token-based, decoder-only multi-modal language model capable of generating and infilling both text and images. C…
Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?
Ari Holtzman, Peter West, Luke Zettlemoyer
Coaxing out desired behavior from pretrained models, while avoiding undesirable ones, has redefined NLP and is reshaping how we interact with computers. What was once a scientific…
QLoRA: Efficient Finetuning of Quantized LLMs
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman +1
We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning t…
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Lili Yu, Dániel Simig, Colin Flaherty +3
Autoregressive transformers are spectacular models for short sequences but scale poorly to long sequences such as high-resolution images, podcasts, code, or books. We proposed Mega…
Stable and low-precision training for large-scale vision-language models
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer +3
We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models. 1) For acceleration, we introduce SwitchBack, a linear layer for int8 qua…
AGRO: Adversarial Discovery of Error-prone groups for Robust Optimization
Bhargavi Paranjape, Pradeep Dasigi, Vivek Srikumar +2
Models trained via empirical risk minimization (ERM) are known to rely on spurious correlations between labels and task-independent input features, resulting in poor generalization…