249 citations · 798 across the 17 of their papers we have counts for
20 papers
Revisiting [CLS] and Patch Token Interaction in Vision Transformers
Alexis Marouani, Oriane Siméoni, Hervé Jégou +2
Vision Transformers have emerged as powerful, scalable and versatile representation learners. To capture both global and local features, a learnable [CLS] class token is typically…
Co-training Submodels for Visual Recognition
Hugo Touvron, Matthieu Cord, Maxime Oquab +3
We introduce submodel co-training, a regularization method related to co-training, self-distillation and stochastic depth. Given a neural network to be trained, for each sample we…
Active Image Indexing
Pierre Fernandez, Matthijs Douze, Hervé Jégou +1
Image copy detection and retrieval from large databases leverage two components. First, a neural network maps an image to a vector representation, that is relatively robust to vari…
DeiT III: Revenge of the ViT
Hugo Touvron, Matthieu Cord, Hervé Jégou
A Vision Transformer (ViT) is a simple neural architecture amenable to serve several computer vision tasks. It has limited built-in architectural priors, in contrast to more recent…
Three things everyone should know about Vision Transformers
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby +2
After their initial success in natural language processing, transformer architectures have rapidly gained traction in computer vision, providing state-of-the-art results for tasks…
Augmenting Convolutional networks with attention-based aggregation
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby +4
We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning. We replace the final average pooling by an attention-based aggre…
Watermarking Images in Self-Supervised Latent Spaces
Pierre Fernandez, Alexandre Sablayrolles, Teddy Furon +2
We revisit watermarking techniques based on pre-trained deep networks, in the light of self-supervised approaches. We present a way to embed both marks and binary messages into the…
Nearest neighbor search with compact codes: A decoder perspective
Kenza Amara, Matthijs Douze, Alexandre Sablayrolles +1
Modern approaches for fast retrieval of similar vectors on billion-scaled datasets rely on compressed-domain approaches such as binary sketches or product quantization. These metho…
ResNet strikes back: An improved training procedure in timm
Ross Wightman, Hugo Touvron, Hervé Jégou
The influential Residual Networks designed by He et al. remain the gold-standard architecture in numerous scientific publications. They typically serve as the default architecture…
ResMLP: Feedforward networks for image classification with data-efficient training
Hugo Touvron, Piotr Bojanowski, Mathilde Caron +8
We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. It is a simple residual network that alternates (i) a linear layer in which…
Emerging Properties in Self-Supervised Vision Transformers
Mathilde Caron, Hugo Touvron, Ishan Misra +4
In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the…
LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron +4
We design a family of image classification architectures that optimize the trade-off between accuracy and efficiency in a high-speed regime. Our work exploits recent findings in at…
Gradient-based Adversarial Attacks against Text Transformers
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou +1
We propose the first general-purpose gradient-based attack against transformer models. Instead of searching for a single adversarial example, we search for a distribution of advers…
Going deeper with Image Transformers
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles +2
Transformers have been recently adapted for large scale image classification, achieving high scores shaking up the long supremacy of convolutional neural networks. However the opti…
Training Vision Transformers for Image Retrieval
Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev +1
Transformers have shown outstanding results for natural language understanding and, more recently, for image classification. We here extend this work and propose a transformer-base…
Training data-efficient image transformers & distillation through attention
Hugo Touvron, Matthieu Cord, Matthijs Douze +3
Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained…
Powers of layers for image-to-image translation
Hugo Touvron, Matthijs Douze, Matthieu Cord +1
We propose a simple architecture to address unpaired image-to-image translation tasks: style or class transfer, denoising, deblurring, deblocking, etc. We start from an image autoe…
Fixing the train-test resolution discrepancy: FixEfficientNet
Hugo Touvron, Andrea Vedaldi, Matthijs Douze +1
This paper provides an extensive analysis of the performance of the EfficientNet image classifiers with several recent training procedures, in particular one that corrects the disc…
Radioactive data: tracing through training
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid +1
We want to detect whether a particular image dataset has been used to train a model. We propose a new technique, \emph{radioactive data}, that makes imperceptible changes to this d…
Word Translation Without Parallel Data
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato +2
State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data…