310 citations · 388 across the 5 of their papers we have counts for
8 papers
Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
Pierre Ablin, Angelos Katharopoulos, Skyler Seto +1
Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-…
Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks
Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger +1
Impressive progress in 3D shape extraction led to representations that can capture object geometries with high fidelity. In parallel, primitive-based methods seek to represent obje…
Fast Transformers with Clustered Attention
Apoorv Vyas, Angelos Katharopoulos, François Fleuret
Transformers have been proven a successful model for a variety of tasks in sequence modeling. However, computing the attention matrix, which is their key component, has quadratic c…
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas +1
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long se…
Processing Megapixel Images with Deep Attention-Sampling Models
Angelos Katharopoulos, François Fleuret
Existing deep architectures cannot operate on very large signals such as megapixel images due to computational and memory constraints. To tackle this limitation, we propose a fully…
Not All Samples Are Created Equal: Deep Learning with Importance Sampling
Angelos Katharopoulos, François Fleuret
Deep neural network training spends most of the computation on examples that are properly handled, and could be ignored. We propose to mitigate this phenomenon with a principled im…