45 citations · 121 across the 15 of their papers we have counts for
12 papers · 1 filter
Understanding Prompt Tuning and In-Context Learning via Meta-Learning
Tim Genewein, Li Kevin Wenliang, Jordi Grau-Moya +3
Prompting is one of the main ways to adapt a pretrained model to target tasks. Besides manually constructing prompts, many prompt optimization methods have been proposed in the lit…
Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
David Heurtel-Depeiges, Anian Ruoss, Joel Veness +1
Foundation models are strong data compressors, but when accounting for their parameter size, their compression ratios are inferior to standard compression algorithms. Naively reduc…
Amortized Planning with Large-Scale Transformers: A Case Study on Chess
Anian Ruoss, Grégoire Delétang, Sourabh Medapati +7
This paper uses chess, a landmark planning problem in AI, to assess transformers' performance on a planning task where memorization is futile $\unicode{x2013}$ even at a large scal…
Learning Universal Predictors
Jordi Grau-Moya, Tim Genewein, Marcus Hutter +8
Meta-learning has emerged as a powerful approach to train neural networks to learn new tasks quickly from limited data. Broad exposure to different tasks leads to versatile represe…
Language Modeling Is Compression
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne +9
It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has f…
Randomized Positional Encodings Boost Length Generalization of Transformers
Anian Ruoss, Grégoire Delétang, Tim Genewein +5
Transformers have impressive generalization capabilities on tasks with a fixed context length. However, they fail to generalize to sequences of arbitrary length, even for seemingly…
Memory-Based Meta-Learning on Non-Stationary Distributions
Tim Genewein, Grégoire Delétang, Anian Ruoss +7
Memory-based meta-learning is a technique for approximating Bayes-optimal predictors. Under fairly general conditions, minimizing sequential prediction error, measured by the log l…
Neural Networks and the Chomsky Hierarchy
Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya +8
Reliable generalization lies at the heart of safe ML and AI. However, understanding when and how neural networks generalize remains one of the most important unsolved problems in t…
Fair Normalizing Flows
Mislav Balunović, Anian Ruoss, Martin Vechev
Fair representation learning is an attractive approach that promises fairness of downstream predictors by encoding sensitive data. Unfortunately, recent work has shown that strong…
Robustness Certification for Point Cloud Models
Tobias Lorenz, Anian Ruoss, Mislav Balunović +2
The use of deep 3D point cloud models in safety-critical applications, such as autonomous driving, dictates the need to certify the robustness of these models to real-world transfo…
Efficient Certification of Spatial Robustness
Anian Ruoss, Maximilian Baader, Mislav Balunović +1
Recent work has exposed the vulnerability of computer vision models to vector field attacks. Due to the widespread usage of such models in safety-critical applications, it is cruci…
Learning Certified Individually Fair Representations
Anian Ruoss, Mislav Balunović, Marc Fischer +1
Fair representation learning provides an effective way of enforcing fairness constraints without compromising utility for downstream users. A desirable family of such fairness cons…