Compositional Kronecker Context Optimization for Vision-Language Models

Abstract

Context Optimization (CoOp) has emerged as a simple yet effective technique for adapting CLIP-like vision-language models to downstream image recognition tasks. Nevertheless, learning compact context with satisfactory base-to-new, domain and cross-task generalization ability while adapting to new tasks is still a challenge. To tackle such a challenge, we propose a lightweight yet generalizable approach termed Compositional Kronecker Context Optimization (CK-CoOp). Technically, the prompt's context words in CK-CoOp are learnable vectors, which are crafted by linearly combining base vectors sourced from a dictionary. These base vectors consist of a non-learnable component obtained by quantizing the weights in the token embedding layer, and a learnable component constructed by applying Kronecker product on several learnable tiny matrices. Intuitively, the compositional structure mitigates the risk of overfitting on training data by remembering more pre-trained knowledge. Meantime, the Kronecker product breaks the non-learnable restrictions of the dictionary, thereby enhancing representation ability with minimal additional parameters. Extensive experiments confirm that CK-CoOp achieves state-of-the-art performance under base-to-new, domain and cross-task generalization evaluation, but also has the metrics of fewer learnable parameters and efficient training and inference speed.

References in corpus (10)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Learning to Prompt for Vision-Language Models
• Fine-Grained Visual Classification of Aircraft
• Florence: A New Foundation Model for Computer Vision
• CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices
• Self-Training Vision Language BERTs with a Unified Conditional Model
• VL-BEiT: Generative Vision-Language Pretraining
• Prompt Tuning with Soft Context Sharing for Vision-Language Models
• Structured Prompt Tuning