570 citations · 868 across the 31 of their papers we have counts for
32 papers
Gemma 4 Technical Report
Gemma Team, Sherif El Abd, Vaibhav Aggarwal +320
We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemm…
Understanding Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Models
Xiangming Gu, Soham De, Larisa Markeeva +2
Large Reasoning Models (LRMs) have shown remarkable performance on challenging questions, such as math and coding. However, to obtain a high quality solution, one may need to sampl…
The Illusion of Stochasticity in LLMs
Xiangming Gu, Soham De, Michalis Titsias +3
In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents. Agentic systems ar…
Mining Generalizable Activation Functions
Alex Vitvitskyi, Michael Boratko, Matej Grcic +3
The choice of activation function is an active area of research, with different proposals aimed at improving optimization, while maintaining expressivity. Additionally, the activat…
Perplexity Cannot Always Tell Right from Wrong
Petar Veličković, Federico Barbero, Christos Perivolaropoulos +2
Perplexity -- a function measuring a model's overall level of "surprise" when encountering a particular output -- has gained significant traction in recent years, both as a loss fu…
Rotary Position Encodings for Graphs
Isaac Reid, Arijit Sehanobish, Cederik Höfs +7
We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transform…
KNARsack: Teaching Neural Algorithmic Reasoners to Solve Pseudo-Polynomial Problems
Stjepan Požgaj, Dobrik Georgiev, Marin Šilić +1
Neural algorithmic reasoning (NAR) is a growing field that aims to embed algorithmic logic into neural networks by imitating classical algorithms. In this extended abstract, we det…
Mastering Board Games by External and Internal Planning with Language Models
John Schultz, Jakub Adamek, Matej Jusup +13
Advancing planning and reasoning capabilities of Large Language Models (LLMs) is one of the key prerequisites towards unlocking their potential for performing reliably in complex a…
Round and Round We Go! What makes Rotary Positional Encodings useful?
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos +2
Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position informatio…
Positional Attention: Expressivity and Learnability of Algorithmic Computation
Artur Back de Luca, George Giapitzakis, Shenghao Yang +2
There is a growing interest in the ability of neural networks to execute algorithmic tasks (e.g., arithmetic, summary statistics, and sorting). The goal of this work is to better u…
Softmax is not Enough (for Sharp Size Generalisation)
Petar Veličković, Christos Perivolaropoulos, Federico Barbero +1
A key property of reasoning systems is the ability to make sharp decisions on their input data. For contemporary AI systems, a key carrier of sharp behaviour is the softmax functio…
Transformers meet Neural Algorithmic Reasoners
Wilfried Bounsi, Borja Ibarz, Andrew Dudzik +5
Transformers have revolutionized machine learning with their simple yet effective architecture. Pre-training Transformers on massive text datasets from the Internet has led to unma…
The CLRS-Text Algorithmic Reasoning Language Benchmark
Larisa Markeeva, Sean McLeish, Borja Ibarz +7
Eliciting reasoning capabilities from language models (LMs) is a critical direction on the path towards building intelligent systems. Most recent studies dedicated to reasoning foc…
Transformers need glasses! Information over-squashing in language tasks
Federico Barbero, Andrea Banino, Steven Kapturowski +5
We study how information propagates in decoder-only Transformers, which are the architectural backbone of most existing frontier large language models (LLMs). We rely on a theoreti…
Finding Increasingly Large Extremal Graphs with AlphaZero and Tabu Search
Abbas Mehrabian, Ankit Anand, Hyunjik Kim +16
This work studies a central extremal graph theory problem inspired by a 1975 conjecture of Erdős, which aims to find graphs with a given size (number of nodes) that maximize the nu…
Neural Priority Queues for Graph Neural Networks
Rishabh Jain, Petar Veličković, Pietro Liò
Graph Neural Networks (GNNs) have shown considerable success in neural algorithmic reasoning. Many traditional algorithms make use of an explicit memory in the form of a data struc…
Asynchronous Algorithmic Alignment with Cocycles
Andrew Dudzik, Tamara von Glehn, Razvan Pascanu +1
State-of-the-art neural algorithmic reasoners make use of message passing in graph neural networks (GNNs). But typical GNNs blur the distinction between the definition and invocati…
How does over-squashing affect the power of GNNs?
Francesco Di Giovanni, T. Konstantin Rusch, Michael M. Bronstein +4
Graph Neural Networks (GNNs) are the state-of-the-art model for machine learning on graph-structured data. The most popular class of GNNs operate by exchanging information between…
Neural Algorithmic Reasoning with Causal Regularisation
Beatrice Bevilacqua, Kyriacos Nikiforou, Borja Ibarz +5
Recent work on neural algorithmic reasoning has investigated the reasoning capabilities of neural networks, effectively demonstrating they can learn to execute classical algorithms…
Learning Graph Search Heuristics
Michal Pándy, Weikang Qiu, Gabriele Corso +4
Searching for a path between two nodes in a graph is one of the most well-studied and fundamental problems in computer science. In numerous domains such as robotics, AI, or biology…
Continuous Neural Algorithmic Planners
Yu He, Petar Veličković, Pietro Liò +1
Neural algorithmic reasoning studies the problem of learning algorithms with neural networks, especially with graph architectures. A recent proposal, XLVIN, reaps the benefits of u…
Neural Algorithmic Reasoners are Implicit Planners
Andreea Deac, Petar Veličković, Ognjen Milinković +3
Implicit planning has emerged as an elegant technique for combining learned models of the world with end-to-end model-free reinforcement learning. We study the class of implicit pl…
Neural Distance Embeddings for Biological Sequences
Gabriele Corso, Rex Ying, Michal Pándy +3
The development of data-dependent heuristics and representations for biological sequences that reflect their evolutionary distance is critical for large-scale biological research.…
ETA Prediction with Graph Neural Networks in Google Maps
Austin Derrow-Pinion, Jennifer She, David Wong +14
Travel-time prediction constitutes a task of high importance in transportation networks, with web mapping services like Google Maps regularly serving vast quantities of travel time…