6 papers
A Mechanistic Analysis of Looped Reasoning Language Models
Hugh Blayney, Ãlvaro Arroyo, Johan Obando-Ceron +4
Reasoning has become a central capability in large language models. Recent research has shown that reasoning performance can be improved by looping an LLM's layers in the latent di…
Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
Enrique Queipo-de-Llano, Ãlvaro Arroyo, Federico Barbero +4
Attention sinks and compression valleys have attracted significant attention as two puzzling phenomena in large language models, but have been studied in isolation. In this work, w…
On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning
Ãlvaro Arroyo, Alessio Gravina, Benjamin Gutteridge +5
Graph Neural Networks (GNNs) are models that leverage the graph structure to transmit information between nodes, typically through the message-passing operation. While widely succe…
Return of ChebNet: Understanding and Improving an Overlooked GNN on Long Range Tasks
Ali Hariri, Ãlvaro Arroyo, Alessio Gravina +6
ChebNet, one of the earliest spectral GNNs, has largely been overshadowed by Message Passing Neural Networks (MPNNs), which gained popularity for their simplicity and effectiveness…
gLSTM: Mitigating Over-Squashing by Increasing Storage Capacity
Hugh Blayney, Ãlvaro Arroyo, Xiaowen Dong +1
Graph Neural Networks (GNNs) leverage the graph structure to transmit information between nodes, typically through the message-passing mechanism. While these models have found a wi…
Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching
Fernando Moreno-Pino, Ãlvaro Arroyo, Harrison Waldon +2
Time-series data in real-world settings typically exhibit long-range dependencies and are observed at non-uniform intervals. In these settings, traditional sequence-based recurrent…