Face-Cap: Image Captioning using Facial Expression Analysis
arXiv:1807.02250
Abstract
Image captioning is the process of generating a natural language description of an image. Most current image captioning models, however, do not take into account the emotional aspect of an image, which is very relevant to activities and interpersonal relationships represented therein. Towards developing a model that can produce human-like captions incorporating these, we use facial expression features extracted from images including human faces, with the aim of improving the descriptive ability of the model. In this work, we present two variants of our Face-Cap model, which embed facial expression features in different ways, to generate image captions. Using all standard evaluation metrics, our Face-Cap models outperform a state-of-the-art baseline model for generating image captions when applied to an image caption dataset extracted from the standard Flickr 30K dataset, consisting of around 11K images containing faces. An analysis of the captions finds that, perhaps surprisingly, the improvement in caption quality appears to come not from the addition of adjectives linked to emotional aspects of the images, but from more variety in the actions described in the captions.
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Sequence to Sequence Learning with Neural Networks
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Deep Learning using Linear Support Vector Machines
- Facial Expression Recognition using Convolutional Neural Networks: State of the Art
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- Image Captioning at Will: A Versatile Scheme for Effectively Injecting Sentiments into Image Descriptions