4 papers
On the "Induction Bias" in Sequence Models
M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal +1
Despite the remarkable practical success of transformer-based language models, recent work has raised concerns about their ability to perform state tracking. In particular, a growi…
Revisiting Bi-Linear State Transitions in Recurrent Neural Networks
M. Reza Ebrahimi, Roland Memisevic
The role of hidden units in recurrent neural networks is typically seen as modeling memory, with research focusing on enhancing information retention through gating mechanisms. A l…
Replacing thinking with tool usage enables reasoning in small language models
Corrado Rainone, Tim Bakker, Roland Memisevic
Recent advances have established a new machine learning paradigm based on scaling up compute at inference time as well as at training time. In that line of work, a combination of S…
Multi-Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits
Ashish Khisti, M. Reza Ebrahimi, Hassan Dbouk +3
We consider multi-draft speculative sampling, where the proposal sequences are sampled independently from different draft models. At each step, a token-level draft selection scheme…