CenterCLIP: Token Clustering for Efficient Text-Video Retrieval

Abstract

Recently, large-scale pre-training methods like CLIP have made great progress in multi-modal research such as text-video retrieval. In CLIP, transformers are vital for modeling complex multi-modal relations. However, in the vision transformer of CLIP, the essential visual tokenization process, which produces discrete visual token sequences, generates many homogeneous tokens due to the redundancy nature of consecutive and similar frames in videos. This significantly increases computation costs and hinders the deployment of video retrieval models in web applications. In this paper, to reduce the number of redundant video tokens, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones. As the frame redundancy occurs mostly in consecutive frames, we divide videos into multiple segments and conduct segment-level clustering. Center tokens from each segment are later concatenated into a new sequence, while their original spatial-temporal relations are well maintained. We instantiate two clustering algorithms to efficiently find deterministic medoids and iteratively partition groups in high dimensional space. Through this token clustering and center selection procedure, we successfully reduce computation costs by removing redundant visual tokens. This method further enhances segment-level semantic alignment between video and text representations, enforcing the spatio-temporal interactions of tokens from within-segment frames. Our method, coined as CenterCLIP, surpasses existing state-of-the-art by a large margin on typical text-video benchmarks, while reducing the training memory cost by 35\% and accelerating the inference speed by 14\% at the best case. The code is available at \href{https://github.com/mzhaoshuai/CenterCLIP}{https://github.com/mzhaoshuai/CenterCLIP}.

arXiv commentaccepted by SIGIR 2022, code is at https://github.com/mzhaoshuai/CenterCLIP

References in corpus (19)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decoupled Weight Decay Regularization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On the Opportunities and Risks of Foundation Models
• Longformer: The Long-Document Transformer
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Mixed Precision Training
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Linformer: Self-Attention with Linear Complexity
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Florence: A New Foundation Model for Computer Vision
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
• Choose a Transformer: Fourier or Galerkin
• AdaViT: Adaptive Tokens for Efficient Vision Transformer

Cited by in corpus (9)

• CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model
• Dual-Modal Attention-Enhanced Text-Video Retrieval with Triplet Partial Margin Contrastive Learning
• Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language
• RTQ: Rethinking Video-language Understanding Based on Image-text Model
• T2VIndexer: A Generative Video Indexer for Efficient Text-Video Retrieval
• WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary Knowledge
• PRVR: Partially Relevant Video Retrieval
• Combating Label Noise With A General Surrogate Model For Sample Selection
• TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrieval