103 citations · 141 across the 19 of their papers we have counts for
4 papers · 1 filter
Subjective Depth and Timescale Transformers: Learning Where and When to Compute
Frederico Wieser, Martin Benfeghoul, Haitham Bou Ammar +2
The rigid, uniform allocation of computation in standard Transformer (TF) architectures can limit their efficiency and scalability, particularly for large-scale models and long seq…
Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods
Martin Benfeghoul, Teresa Delgado, Adnan Oomerjee +3
Transformers' quadratic computational complexity limits their scalability despite remarkable performance. While linear attention reduces this to linear complexity, pre-training suc…
Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective
Matthieu Zimmer, Xiaotong Ji, Tu Nguyen +1
We introduce a novel approach to large language model (LLM) distillation by formulating it as a constrained reinforcement learning problem. While recent work has begun exploring th…
On Almost Surely Safe Alignment of Large Language Models at Inference-Time
Xiaotong Ji, Shyam Sundhar Ramesh, Matthieu Zimmer +3
We introduce a novel inference-time alignment approach for LLMs that aims to generate safe responses almost surely, i.e., with probability approaching one. Our approach models the…