The Unreasonable Effectiveness of Large Language-Vision Models for Source-free Video Domain Adaptation

Abstract

Source-Free Video Unsupervised Domain Adaptation (SFVUDA) task consists in adapting an action recognition model, trained on a labelled source dataset, to an unlabelled target dataset, without accessing the actual source data. The previous approaches have attempted to address SFVUDA by leveraging self-supervision (e.g., enforcing temporal consistency) derived from the target data itself. In this work, we take an orthogonal approach by exploiting "web-supervision" from Large Language-Vision Models (LLVMs), driven by the rationale that LLVMs contain a rich world prior surprisingly robust to domain-shift. We showcase the unreasonable effectiveness of integrating LLVMs for SFVUDA by devising an intuitive and parameter-efficient method, which we name Domain Adaptation with Large Language-Vision models (DALL-V), that distills the world prior and complementary source model information into a student network tailored for the target. Despite the simplicity, DALL-V achieves significant improvement over state-of-the-art SFVUDA methods.

arXiv commentAccepted at ICCV2023, 14 pages, 7 figures, code is available at https://github.com/giaczara/dallv

References in corpus (20)

• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Learning to Prompt for Vision-Language Models
• The Kinetics Human Action Video Dataset
• Learning Transferable Features with Deep Adaptation Networks
• Unsupervised Domain Adaptation by Backpropagation
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Model Adaptation: Unsupervised Domain Adaptation without Source Data
• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• ActionCLIP: A New Paradigm for Video Action Recognition
• Video Transformers: A Survey
• A Comprehensive Study of Deep Video Action Recognition
• Aligning Correlation Information for Domain Adaptation in Action Recognition
• Source-free Domain Adaptation via Avatar Prototype Generation and Adaptation
• Leveraging Endo- and Exo-Temporal Regularization for Black-box Video Domain Adaptation
• Multi-Source Video Domain Adaptation with Temporal Attentive Moment Alignment