Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach
arXiv:1909.02201 · doi:10.18653/v1/D19-1208
Abstract
Constructing an organized dataset comprised of a large number of images and several captions for each image is a laborious task, which requires vast human effort. On the other hand, collecting a large number of images and sentences separately may be immensely easier. In this paper, we develop a novel data-efficient semi-supervised framework for training an image captioning model. We leverage massive unpaired image and caption data by learning to associate them. To this end, our proposed semi-supervised learning method assigns pseudo-labels to unpaired samples via Generative Adversarial Networks to learn the joint distribution of image and caption. To evaluate, we construct scarcely-paired COCO dataset, a modified version of MS COCO caption dataset. The empirical results show the effectiveness of our method compared to several strong baselines, especially when the amount of the paired samples are scarce.
arXiv commentEMNLP 2019. Project page : https://sites.google.com/view/emnlp19scarcecaption
References in corpus (22)
- Adam: A Method for Stochastic Optimization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Character-level Convolutional Networks for Text Classification
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
- Semi-Supervised Learning with Deep Generative Models
- Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
- Unsupervised Image-to-Image Translation Networks
- Phrase-Based & Neural Unsupervised Machine Translation
- Triple Generative Adversarial Nets
- Unsupervised Machine Translation Using Monolingual Corpora Only
- Unsupervised Neural Machine Translation
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning
- Triangle Generative Adversarial Networks
- CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise
- Iterative Learning with Open-set Noisy Labels
- Joint Training for Neural Machine Translation Models with Monolingual Data
- Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
- Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data
- Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
- Unpaired Image Captioning by Language Pivoting
- Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning
- Disjoint Multi-task Learning between Heterogeneous Human-centric Tasks