6 papers
From AGI to ASI
Tim Genewein, Matija Franklin, Alexander Lerchner +11
Over the last decade, building human-level artificial general intelligence has moved from far-fetched speculation to being a concrete next-decade target for many of the largest AI…
Why is prompting hard? Understanding prompts on binary sequence predictors
Li Kevin Wenliang, Anian Ruoss, Jordi Grau-Moya +2
Frontier models can be prompted or conditioned to do many tasks, but finding good prompts is not always easy, nor is understanding some performant prompts. We view prompting as fin…
Understanding Prompt Tuning and In-Context Learning via Meta-Learning
Tim Genewein, Li Kevin Wenliang, Jordi Grau-Moya +3
Prompting is one of the main ways to adapt a pretrained model to target tasks. Besides manually constructing prompts, many prompt optimization methods have been proposed in the lit…
LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Anian Ruoss, Fabio Pardo, Harris Chan +3
In this paper, we present a benchmark to pressure-test today's frontier models' multimodal decision-making capabilities in the very long-context regime (up to one million tokens) a…
Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
David Heurtel-Depeiges, Anian Ruoss, Joel Veness +1
Foundation models are strong data compressors, but when accounting for their parameter size, their compression ratios are inferior to standard compression algorithms. Naively reduc…
Amortized Planning with Large-Scale Transformers: A Case Study on Chess
Anian Ruoss, Grégoire Delétang, Sourabh Medapati +7
This paper uses chess, a landmark planning problem in AI, to assess transformers' performance on a planning task where memorization is futile $\unicode{x2013}$ even at a large scal…