6 papers
RotRNN: Modelling Long Sequences with Rotations
Kai Biegun, Rares Dolga, Jake Cunningham +1
Linear recurrent neural networks, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling b…
wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models
Xiaohang Tang, Rares Dolga, Sangwoong Yoon +1
Improving the reasoning capabilities of diffusion-based large language models (dLLMs) through reinforcement learning (RL) remains an open problem. The intractability of dLLMs likel…
Unifying Linear-Time Attention via Latent Probabilistic Modelling
Rares Dolga, Lucas Maystre, Marius Cobzarenco +1
Transformers have achieved state-of-the-art results across a range of domains, but their quadratic attention mechanism poses significant challenges for long-sequence modelling. Rec…
Incremental Sequence Classification with Temporal Consistency
Lucas Maystre, Gabriel Barello, Tudor Berariu +5
We address the problem of incremental sequence classification, where predictions are updated as new elements in the sequence are revealed. Drawing on temporal-difference learning f…
From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
Rares Dolga, Lucas Maystre, Tudor Berariu +1
Subword tokenization methods like Byte Pair Encoding (BPE) are widely used in large language models due to their balance of vocabulary compactness and representational power. Howev…
When Embedding Models Meet: Procrustes Bounds and Applications
Lucas Maystre, Alvaro Ortega Gonzalez, Charles Park +4
Embedding models trained separately on similar data often produce representations that encode stable information but are not directly interchangeable. This lack of interoperability…