3 citations · 3 across the 17 of their papers we have counts for
19 papers
Across the Loss Landscape with Progressive Growth
Paul Caillon, Christophe Cerisara, Alexandre Allauzen
Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geometry of the minima found by stoch…
Higher-Order Fourier Neural Operator: Explicit Mode Mixer for Nonlinear PDEs
Alex Colagrande, Paul Caillon, Eva Feillet +1
Neural operators provide deep neural networks for learning mappings between function spaces. Among them, the Fourier Neural Operator (FNO) is particularly effective: its spectral c…
Limits of Resolution Equivariance in Fourier Neural Operators
Alex Colagrande, Paul Caillon, Eva Feillet +1
Fourier Neural Operators are often assumed to generalize across spatial resolutions, enabling training on a coarse grid and deployment on a finer grid. We test this assumption by c…
Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
Erwan Fagnou, Paul Caillon, Blaise Delattre +1
Token mixing layers play a key role in how language models can learn and generate long-range dependencies. Their efficiency relies on the necessary trade-off between decoding speed…
Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
Hangyue Zhao, Paul Caillon, Erwan Fagnou +1
Entity tracking requires maintaining and updating latent states for entities and attributes over long sequences. Recent task-specific attention operators can compress deep Transfor…
Forward Only Learning for Orthogonal Neural Networks of any Depth
Paul Caillon, Alex Colagrande, Erwan Fagnou +2
Backpropagation is still the de facto algorithm used today to train neural networks. With the exponential growth of recent architectures, the computational cost of this algorithm a…
On the MIA Vulnerability Gap Between Private GANs and Diffusion Models
Ilana Sebag, Jean-Yves Franceschi, Alain Rakotomamonjy +2
Generative Adversarial Networks (GANs) and diffusion models have emerged as leading approaches for high-quality image synthesis. While both can be trained under differential privac…
Improving Diversity in Language Models: When Temperature Fails, Change the Loss
Alexandre Verine, Florian Le Bronnec, Kunhao Zheng +3
Increasing diversity in language models is a challenging yet essential objective. A common approach is to raise the decoding temperature. In this work, we investigate this approach…
Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physics
Alex Colagrande, Paul Caillon, Eva Feillet +1
Transformers have become the de facto standard for a wide range of tasks, from image classification to physics simulations. Despite their impressive performance, the quadratic comp…
Bridging the Theoretical Gap in Randomized Smoothing
Blaise Delattre, Paul Caillon, Quentin Barthélemy +2
Randomized smoothing has become a leading approach for certifying adversarial robustness in machine learning models. However, a persistent gap remains between theoretical certified…
Fast Training of Recurrent Neural Networks with Stationary State Feedbacks
Paul Caillon, Erwan Fagnou, Alexandre Allauzen
Recurrent neural networks (RNNs) have recently demonstrated strong performance and faster inference than Transformers at comparable parameter budgets. However, the recursive gradie…
SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text Generation
Song Duong, Florian Le Bronnec, Alexandre Allauzen +4
Large Language Models (LLMs), when used for conditional text generation, often produce hallucinations, i.e., information that is unfaithful or not grounded in the input context. Th…
Conditional Distribution Quantization in Machine Learning
Blaise Delattre, Sylvain Delattre, Alexandre Vérine +1
Conditional expectation \mathbb{E}(Y \mid X) often fails to capture the complexity of multimodal conditional distributions \mathcal{L}(Y \mid X). To address this, we propose using…
Accelerated Training through Iterative Gradient Propagation Along the Residual Path
Erwan Fagnou, Paul Caillon, Blaise Delattre +1
Despite being the cornerstone of deep learning, backpropagation is criticized for its inherent sequentiality, which can limit the scalability of very deep models. Such models faced…
Chain and Causal Attention for Efficient Entity Tracking
Erwan Fagnou, Paul Caillon, Blaise Delattre +1
This paper investigates the limitations of transformers for entity-tracking tasks in large language models. We identify a theoretical constraint, showing that transformers require…
Exploring Precision and Recall to assess the quality and diversity of LLMs
Florian Le Bronnec, Alexandre Verine, Benjamin Negrevergne +2
We introduce a novel evaluation framework for Large Language Models (LLMs) such as \textsc{Llama-2} and \textsc{Mistral}, focusing on importing Precision and Recall metrics from im…
Spectral Norm of Convolutional Layers with Circular and Zero Paddings
Blaise Delattre, Quentin Barthélemy, Alexandre Allauzen
This paper leverages the use of \emph{Gram iteration} an efficient, deterministic, and differentiable method for computing spectral norm with an upper bound guarantee. Designed for…
LOCOST: State-Space Models for Long Document Abstractive Summarization
Florian Le Bronnec, Song Duong, Mathieu Ravaut +6
State-space models are a low-complexity alternative to transformers for encoding long sequences and capturing long-term dependencies. We propose LOCOST: an encoder-decoder architec…
Differentially Private Gradient Flow based on the Sliced Wasserstein Distance
Ilana Sebag, Muni Sreenivas Pydi, Jean-Yves Franceschi +4
Safeguarding privacy in sensitive training data is paramount, particularly in the context of generative modeling. This can be achieved through either differentially private stochas…