1.1k citations · 1.7k across the 21 of their papers we have counts for
19 papers · 1 filter
Scaling Laws for Native Multimodal Models
Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa +3
Building general-purpose models that can effectively perceive the world through multimodal signals has been a long-standing goal. Current approaches involve integrating separately…
FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
Roman Bachmann, Jesse Allardice, David Mizrahi +6
Image tokenization has enabled major advances in autoregressive image generation by providing compressed, discrete representations that are more efficient to process than raw pixel…
Multimodal Autoregressive Pre-training of Large Vision Encoders
Enrico Fini, Mustafa Shukor, Xiujun Li +13
We introduce a novel method for pre-training of large-scale vision encoders. Building on recent advancements in autoregressive pre-training of vision models, we extend this framewo…
Scalable Pre-training of Large Autoregressive Image Models
Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai +5
This paper introduces AIM, a collection of vision models pre-trained with an autoregressive objective. These models are inspired by their textual counterparts, i.e., Large Language…
ImageBind: One Embedding Space To Bind Them All
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu +4
We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of p…
Are Visual Recognition Models Robust to Image Compression?
João Maria Janeiro, Stanislav Frolov, Alaaeldin El-Nouby +1
Reducing the data footprint of visual content via image compression is essential to reduce storage requirements, but also to reduce the bandwidth and latency requirements for trans…
DINOv2: Learning Robust Visual Features without Supervision
Maxime Oquab, Timothée Darcet, Théo Moutakanni +23
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. Thes…
Image Compression with Product Quantized Masked Image Modeling
Alaaeldin El-Nouby, Matthew J. Muckley, Karen Ullrich +3
Recent neural compression methods have been based on the popular hyperprior framework. It relies on Scalar Quantization and offers a very strong compression performance. This contr…
OmniMAE: Single Model Masked Pretraining on Images and Videos
Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh +3
Transformer-based architectures have become competitive across a variety of visual domains, most notably images and videos. While prior work studies these modalities in isolation,…
Three things everyone should know about Vision Transformers
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby +2
After their initial success in natural language processing, transformer architectures have rapidly gained traction in computer vision, providing state-of-the-art results for tasks…
Augmenting Convolutional networks with attention-based aggregation
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby +4
We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning. We replace the final average pooling by an attention-based aggre…
Are Large-scale Datasets Necessary for Self-Supervised Pre-training?
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron +3
Pre-training models on large scale datasets, like ImageNet, is a standard practice in computer vision. This paradigm is especially effective for tasks with small training sets, for…
XCiT: Cross-Covariance Image Transformers
Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron +8
Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yiel…
ResMLP: Feedforward networks for image classification with data-efficient training
Hugo Touvron, Piotr Bojanowski, Mathilde Caron +8
We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. It is a simple residual network that alternates (i) a linear layer in which…
LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron +4
We design a family of image classification architectures that optimize the trade-off between accuracy and efficiency in a high-speed regime. Our work exploits recent findings in at…
Training Vision Transformers for Image Retrieval
Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev +1
Transformers have shown outstanding results for natural language understanding and, more recently, for image classification. We here extend this work and propose a transformer-base…
Skip-Clip: Self-Supervised Spatiotemporal Representation Learning by Future Clip Order Ranking
Alaaeldin El-Nouby, Shuangfei Zhai, Graham W. Taylor +1
Deep neural networks require collecting and annotating large amounts of data to train successfully. In order to alleviate the annotation bottleneck, we propose a novel self-supervi…
Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz +5
Conditional text-to-image generation is an active area of research, with many possible applications. Existing research has primarily focused on generating a single image from avail…
Real-Time End-to-End Action Detection with Two-Stream Networks
Alaaeldin El-Nouby, Graham W. Taylor
Two-stream networks have been very successful for solving the problem of action detection. However, prior work using two-stream networks train both streams separately, which preven…