Contrastive Learning for Image Captioning
arXiv:1710.02534
Abstract
Image captioning, a popular topic in computer vision, has achieved substantial progress in recent years. However, the distinctiveness of natural descriptions is often overlooked in previous work. It is closely related to the quality of captions, as distinctive captions are more likely to describe images with their unique aspects. In this work, we propose a new learning method, Contrastive Learning (CL), for image captioning. Specifically, via two constraints formulated on top of a reference model, the proposed method can encourage distinctiveness, while maintaining the overall quality of the generated captions. We tested our method on two challenging datasets, where it improves the baseline model by significant margins. We also showed in our studies that the proposed method is generic and can be used for models with various structures.
accepted to 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA
Cited by in corpus (25)
- Representation Learning for Natural Language Processing
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
- Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs
- A Neural Compositional Paradigm for Image Captioning
- On Distinctive Image Captioning via Comparing and Reweighting
- Rethinking the Reference-based Distinctive Image Captioning
- Adversarial Contrastive Estimation
- Contrastive Attention for Automatic Chest X-ray Report Generation
- Generating Diverse and Meaningful Captions
- Neural Sign Language Translation based on Human Keypoint Estimation
- Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection
- Contrastively Smoothed Class Alignment for Unsupervised Domain Adaptation
- Contrastive Learning for Robust Android Malware Familial Classification
- Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning
- Move Forward and Tell: A Progressive Generator of Video Descriptions
- Dense Relational Image Captioning via Multi-task Triple-Stream Networks
- Unsupervised Stylish Image Description Generation via Domain Layer Norm
- Group-wise Contrastive Learning for Neural Dialogue Generation
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- Group-based Distinctive Image Captioning with Memory Attention
- Learning Compact Reward for Image Captioning
- DIVE: Towards Descriptive and Diverse Visual Commonsense Generation
- Distinctive-attribute Extraction for Image Captioning
- Rethinking the Form of Latent States in Image Captioning
- Semantic bottleneck for computer vision tasks