206 citations · 298 across the 10 of their papers we have counts for
12 papers
Context-Aware Multimodal Pretraining
Karsten Roth, Zeynep Akata, Dima Damen +2
Large-scale multimodal representation learning successfully optimizes for zero-shot transfer at test time. Yet the standard pretraining paradigm (contrastive learning on large amou…
Active Data Curation Effectively Distills Large-Scale Multimodal Models
Vishaal Udandarao, Nikhil Parthasarathy, Muhammad Ferjad Naeem +6
Knowledge distillation (KD) is the de facto standard for compressing large-scale models into smaller ones. Prior works have explored ever more complex KD strategies involving diffe…
A Practitioner's Guide to Continual Multimodal Pretraining
Karsten Roth, Vishaal Udandarao, Sebastian Dziadzio +7
Multimodal foundation models serve numerous applications at the intersection of vision and language. Still, despite being pretrained on extensive data, they become outdated over ti…
Reflecting on the State of Rehearsal-free Continual Learning with Pretrained Models
Lukas Thede, Karsten Roth, Olivier J. Hénaff +2
With the advent and recent ubiquity of foundation models, continual learning (CL) has recently shifted from continual training from scratch to the continual adaptation of pretraine…
Memory Consolidation Enables Long-Context Video Understanding
Ivana Balažević, Yuge Shi, Pinelopi Papalampidi +3
Most transformer-based video encoders are limited to short temporal contexts due to their quadratic complexity. While various attempts have been made to extend this context, this h…
Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model
Karsten Roth, Lukas Thede, Almut Sophia Koepke +3
Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variati…
Three ways to improve feature alignment for open vocabulary detection
Relja Arandjelović, Alex Andonian, Arthur Mensch +3
The core problem in zero-shot open vocabulary detection is how to align visual and text features, so that the detector performs well on unseen classes. Previous approaches train th…
Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methods
Skanda Koppula, Yazhe Li, Evan Shelhamer +5
Self-supervised methods have achieved remarkable success in transfer learning, often achieving the same or better accuracy than supervised pre-training. Most prior work has done so…
Object discovery and representation networks
Olivier J. Hénaff, Skanda Koppula, Evan Shelhamer +5
The promise of self-supervised learning (SSL) is to leverage large amounts of unlabeled data to solve complex tasks. While there has been excellent progress with simple, image-leve…
Perceiver IO: A General Architecture for Structured Inputs & Outputs
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac +12
A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible. Current architectures, however, cannot be applied…
Efficient Visual Pretraining with Contrastive Detection
Olivier J. Hénaff, Skanda Koppula, Jean-Baptiste Alayrac +3
Self-supervised pretraining has been shown to yield powerful representations for transfer learning. These performance gains come at a large computational cost however, with state-o…
Are we done with ImageNet?
Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov +2
Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overf…