1.1k citations · 3.1k across the 34 of their papers we have counts for
38 papers
Birth of a Transformer: A Memory Viewpoint
Alberto Bietti, Vivien Cabannes, Diane Bouchacourt +2
Large language models based on transformers have achieved great empirical successes. However, as they are deployed more widely, there is a growing need to better understand their i…
DINOv2: Learning Robust Visual Features without Supervision
Maxime Oquab, Timothée Darcet, Théo Moutakanni +23
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. Thes…
Are Large-scale Datasets Necessary for Self-Supervised Pre-training?
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron +3
Pre-training models on large scale datasets, like ImageNet, is a standard practice in computer vision. This paradigm is especially effective for tasks with small training sets, for…
XCiT: Cross-Covariance Image Transformers
Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron +8
Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yiel…
Grafit: Learning fine-grained image representations with coarse labels
Hugo Touvron, Alexandre Sablayrolles, Matthijs Douze +2
This paper tackles the problem of learning a finer representation than the one provided by training labels. This enables fine-grained category retrieval of images in a collection a…
Training with Quantization Noise for Extreme Model Compression
Angela Fan, Pierre Stock, Benjamin Graham +4
We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, wher…
White-box vs Black-box: Bayes Optimal Strategies for Membership Inference
Alexandre Sablayrolles, Matthijs Douze, Yann Ollivier +2
Membership inference determines, given a sample and trained parameters of a machine learning model, whether the sample was part of the training set. In this paper, we derive the op…
Augmenting Self-attention with Persistent Memory
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample +2
Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-att…
And the Bit Goes Down: Revisiting the Quantization of Neural Networks
Pierre Stock, Armand Joulin, Rémi Gribonval +2
In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the…
Large Memory Layers with Product Keys
Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato +2
This paper introduces a structured memory which can be easily integrated into a neural network. The memory is very large by design and significantly increases the capacity of the a…
Fixing the train-test resolution discrepancy
Hugo Touvron, Andrea Vedaldi, Matthijs Douze +1
Data-augmentation is key to the training of neural networks for image classification. This paper first shows that existing augmentations induce a significant discrepancy between th…
Billion-scale semi-supervised learning for image classification
I. Zeki Yalniz, Hervé Jégou, Kan Chen +2
This paper presents a study of semi-supervised learning with large convolutional networks. We propose a pipeline, based on a teacher/student paradigm, that leverages a large collec…
MultiGrain: a unified image embedding for classes and instances
Maxim Berman, Hervé Jégou, Andrea Vedaldi +2
MultiGrain is a network architecture producing compact vector representations that are suited both for image classification and particular object retrieval. It builds on a standard…
Equi-normalization of Neural Networks
Pierre Stock, Benjamin Graham, Rémi Gribonval +1
Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and output weights, wit…
Understanding and Improving Kernel Local Descriptors
Arun Mukundan, Giorgos Tolias, Andrei Bursuc +2
We propose a multiple-kernel local-patch descriptor based on efficient match kernels from pixel gradients. It combines two parametrizations of gradient position and direction, each…
Déjà Vu: an empirical evaluation of the memorization properties of ConvNets
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid +1
Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate overfitting. This paper…
Spreading vectors for similarity search
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid +1
Discretizing multi-dimensional data distributions is a fundamental step of modern indexing methods. State-of-the-art techniques learn parameters of quantizers on training data for…
Link and code: Fast indexing with graphs and compact regression codes
Matthijs Douze, Alexandre Sablayrolles, Hervé Jégou
Similarity search approaches based on graph walks have recently attained outstanding speed-accuracy trade-offs, taking aside the memory requirements. In this paper, we revisit thes…
Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion
Armand Joulin, Piotr Bojanowski, Tomas Mikolov +2
Continuous word representations learned separately on distinct languages can be aligned so that their words become comparable in a common space. Existing works typically solve a le…
An evaluation of large-scale methods for image instance and class discovery
Matthijs Douze, Hervé Jégou, Jeff Johnson
This paper aims at discovering meaningful subsets of related images from large image collections without annotations. We search groups of images related at different levels of sema…
Low-shot learning with large-scale diffusion
Matthijs Douze, Arthur Szlam, Bharath Hariharan +1
This paper considers the problem of inferring image labels from images when only a few annotated examples are available at training time. This setup is often referred to as low-sho…
Billion-scale similarity search with GPUs
Jeff Johnson, Matthijs Douze, Hervé Jégou
Similarity search finds application in specialized database systems handling complex data such as images or videos, which are typically represented by high-dimensional features and…
FastText.zip: Compressing text classification models
Armand Joulin, Edouard Grave, Piotr Bojanowski +3
We consider the problem of producing compact architectures for text classification, such that the full model fits in a limited amount of memory. After considering different solutio…
Interferences in match kernels
Naila Murray, Hervé Jégou, Florent Perronnin +1
We consider the design of an image representation that embeds and aggregates a set of local descriptors into a single vector. Popular representations of this kind include the bag-o…