2 citations · 2 across the 3 of their papers we have counts for
5 papers · 1 filter
Understanding Prompt Tuning and In-Context Learning via Meta-Learning
Tim Genewein, Li Kevin Wenliang, Jordi Grau-Moya +3
Prompting is one of the main ways to adapt a pretrained model to target tasks. Besides manually constructing prompts, many prompt optimization methods have been proposed in the lit…
Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
David Heurtel-Depeiges, Anian Ruoss, Joel Veness +1
Foundation models are strong data compressors, but when accounting for their parameter size, their compression ratios are inferior to standard compression algorithms. Naively reduc…
Amortized Planning with Large-Scale Transformers: A Case Study on Chess
Anian Ruoss, Grégoire Delétang, Sourabh Medapati +7
This paper uses chess, a landmark planning problem in AI, to assess transformers' performance on a planning task where memorization is futile $\unicode{x2013}$ even at a large scal…
Language Modeling Is Compression
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne +9
It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has f…
Learning Universal Predictors
Jordi Grau-Moya, Tim Genewein, Marcus Hutter +8
Meta-learning has emerged as a powerful approach to train neural networks to learn new tasks quickly from limited data. Broad exposure to different tasks leads to versatile represe…