231 citations · 337 across the 12 of their papers we have counts for
12 papers
Gemma 4 Technical Report
Gemma Team, Sherif El Abd, Vaibhav Aggarwal +320
We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemm…
Mining Generalizable Activation Functions
Alex Vitvitskyi, Michael Boratko, Matej Grcic +3
The choice of activation function is an active area of research, with different proposals aimed at improving optimization, while maintaining expressivity. Additionally, the activat…
What makes a good feedforward computational graph?
Alex Vitvitskyi, João G. M. Araújo, Marc Lackenby +1
As implied by the plethora of literature on graph rewiring, the choice of computational graph employed by a neural network can make a significant impact on its downstream performan…
Amplifying human performance in combinatorial competitive programming
Petar Veličković, Alex Vitvitskyi, Larisa Markeeva +4
Recent years have seen a significant surge in complex AI systems for competitive programming, capable of performing at admirable levels against human competitors. While steady prog…
Round and Round We Go! What makes Rotary Positional Encodings useful?
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos +2
Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position informatio…
Transformers meet Neural Algorithmic Reasoners
Wilfried Bounsi, Borja Ibarz, Andrew Dudzik +5
Transformers have revolutionized machine learning with their simple yet effective architecture. Pre-training Transformers on massive text datasets from the Internet has led to unma…
The CLRS-Text Algorithmic Reasoning Language Benchmark
Larisa Markeeva, Sean McLeish, Borja Ibarz +7
Eliciting reasoning capabilities from language models (LMs) is a critical direction on the path towards building intelligent systems. Most recent studies dedicated to reasoning foc…
Transformers need glasses! Information over-squashing in language tasks
Federico Barbero, Andrea Banino, Steven Kapturowski +5
We study how information propagates in decoder-only Transformers, which are the architectural backbone of most existing frontier large language models (LLMs). We rely on a theoreti…
A Generalist Neural Algorithmic Learner
Borja Ibarz, Vitaly Kurin, George Papamakarios +12
The cornerstone of neural algorithmic reasoning is the ability to solve algorithmic tasks, especially in a way that generalises out of distribution. While recent years have seen a…
Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
Víctor Campos, Pablo Sprechmann, Steven Hansen +5
Designing agents that acquire knowledge autonomously and use it to solve new tasks efficiently is an important challenge in reinforcement learning. Knowledge acquired during an uns…
Agent57: Outperforming the Atari Human Benchmark
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski +4
Atari games have been a long-standing benchmark in the reinforcement learning (RL) community for the past decade. This benchmark was proposed to test general competency of RL algor…
Never Give Up: Learning Directed Exploration Strategies
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi +8
We propose a reinforcement learning agent to solve hard exploration games by learning a range of directed exploratory policies. We construct an episodic memory-based intrinsic rewa…