CgT-GAN: CLIP-guided Text GAN for Image Captioning
arXiv:2308.12045 · doi:10.1145/3581783.3611891
Abstract
The large-scale visual-language pre-trained model, Contrastive Language-Image Pre-training (CLIP), has significantly improved image captioning for scenarios without human-annotated image-caption pairs. Recent advanced CLIP-based image captioning without human annotations follows a text-only training paradigm, i.e., reconstructing text from shared embedding space. Nevertheless, these approaches are limited by the training/inference gap or huge storage requirements for text embeddings. Given that it is trivial to obtain images in the real world, we propose CLIP-guided text GAN (CgT-GAN), which incorporates images into the training process to enable the model to "see" real visual modality. Particularly, we use adversarial training to teach CgT-GAN to mimic the phrases of an external text corpus and CLIP-based reward to provide semantic guidance. The caption generator is jointly rewarded based on the caption naturalness to human language calculated from the GAN's discriminator and the semantic guidance reward computed by the CLIP-based reward module. In addition to the cosine similarity as the semantic guidance reward (i.e., CLIP-cos), we further introduce a novel semantic guidance reward called CLIP-agg, which aligns the generated caption with a weighted text embedding by attentively aggregating the entire corpus. Experimental results on three subtasks (ZS-IC, In-UIC and Cross-UIC) show that CgT-GAN outperforms state-of-the-art methods significantly across all metrics. Code is available at https://github.com/Lihr747/CgtGAN.
arXiv commentAccepted at ACM MM 2023
References in corpus (15)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Decoupled Weight Decay Regularization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction
- Improving Image Captioning with Conditional Generative Adversarial Nets
- Language Models Can See: Plugging Visual Controls in Text Generation
- Generative Recommendation: Towards Next-generation Recommender Paradigm
- DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training
- Multimodal Knowledge Alignment with Reinforcement Learning
- I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision