"Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention
arXiv:1807.03871
Abstract
Generating stylized captions for an image is an emerging topic in image captioning. Given an image as input, it requires the system to generate a caption that has a specific style (e.g., humorous, romantic, positive, and negative) while describing the image content semantically accurately. In this paper, we propose a novel stylized image captioning model that effectively takes both requirements into consideration. To this end, we first devise a new variant of LSTM, named style-factual LSTM, as the building block of our model. It uses two groups of matrices to capture the factual and stylized knowledge, respectively, and automatically learns the word-level weights of the two groups based on previous context. In addition, when we train the model to capture stylized elements, we propose an adaptive learning approach based on a reference factual model, it provides factual knowledge to the model as the model learns from stylized caption labels, and can adaptively compute how much information to supply at each time step. We evaluate our model on two stylized image captioning datasets, which contain humorous/romantic captions and positive/negative captions, respectively. Experiments shows that our proposed model outperforms the state-of-the-art approaches, without using extra ground truth supervision.
arXiv comment17 pages, 7 figures, ECCV 2018
References in corpus (14)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Teaching Machines to Read and Comprehend
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Recurrent Models of Visual Attention
- Style Transfer from Non-Parallel Text by Cross-Alignment
- Reasoning about Entailment with Neural Attention
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- Learning Generative Models with Visual Attention
- Review Networks for Caption Generation
- Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
- Rich Image Captioning in the Wild
- Simple Image Description Generator via a Linear Phrase-Based Approach