6 papers
A Mechanistic Analysis of Looped Reasoning Language Models
Hugh Blayney, Ãlvaro Arroyo, Johan Obando-Ceron +4
Reasoning has become a central capability in large language models. Recent research has shown that reasoning performance can be improved by looping an LLM's layers in the latent di…
k-Maximum Inner Product Attention for Graph Transformers and the Expressive Power of GraphGPS
Jonas De Schouwer, Haitz Sáez de Ocáriz Borde, Xiaowen Dong
Graph transformers have shown promise in overcoming limitations of traditional graph neural networks, such as oversquashing and difficulties in modeling long-range dependencies. Ho…
Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
Enrique Queipo-de-Llano, Ãlvaro Arroyo, Federico Barbero +4
Attention sinks and compression valleys have attracted significant attention as two puzzling phenomena in large language models, but have been studied in isolation. In this work, w…
Return of ChebNet: Understanding and Improving an Overlooked GNN on Long Range Tasks
Ali Hariri, Ãlvaro Arroyo, Alessio Gravina +6
ChebNet, one of the earliest spectral GNNs, has largely been overshadowed by Message Passing Neural Networks (MPNNs), which gained popularity for their simplicity and effectiveness…
gLSTM: Mitigating Over-Squashing by Increasing Storage Capacity
Hugh Blayney, Ãlvaro Arroyo, Xiaowen Dong +1
Graph Neural Networks (GNNs) leverage the graph structure to transmit information between nodes, typically through the message-passing mechanism. While these models have found a wi…
Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching
Fernando Moreno-Pino, Ãlvaro Arroyo, Harrison Waldon +2
Time-series data in real-world settings typically exhibit long-range dependencies and are observed at non-uniform intervals. In these settings, traditional sequence-based recurrent…