7 papers
Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
Erwan Fagnou, Paul Caillon, Blaise Delattre +1
Token mixing layers play a key role in how language models can learn and generate long-range dependencies. Their efficiency relies on the necessary trade-off between decoding speed…
Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
Hangyue Zhao, Paul Caillon, Erwan Fagnou +1
Entity tracking requires maintaining and updating latent states for entities and attributes over long sequences. Recent task-specific attention operators can compress deep Transfor…
Forward Only Learning for Orthogonal Neural Networks of any Depth
Paul Caillon, Alex Colagrande, Erwan Fagnou +2
Backpropagation is still the de facto algorithm used today to train neural networks. With the exponential growth of recent architectures, the computational cost of this algorithm a…
Bridging the Theoretical Gap in Randomized Smoothing
Blaise Delattre, Paul Caillon, Quentin Barthélemy +2
Randomized smoothing has become a leading approach for certifying adversarial robustness in machine learning models. However, a persistent gap remains between theoretical certified…
Fast Training of Recurrent Neural Networks with Stationary State Feedbacks
Paul Caillon, Erwan Fagnou, Alexandre Allauzen
Recurrent neural networks (RNNs) have recently demonstrated strong performance and faster inference than Transformers at comparable parameter budgets. However, the recursive gradie…
Accelerated Training through Iterative Gradient Propagation Along the Residual Path
Erwan Fagnou, Paul Caillon, Blaise Delattre +1
Despite being the cornerstone of deep learning, backpropagation is criticized for its inherent sequentiality, which can limit the scalability of very deep models. Such models faced…