A Multi-Modal Approach to Infer Image Affect
arXiv:1803.05070
Abstract
The group affect or emotion in an image of people can be inferred by extracting features about both the people in the picture and the overall makeup of the scene. The state-of-the-art on this problem investigates a combination of facial features, scene extraction and even audio tonality. This paper combines three additional modalities, namely, human pose, text-based tagging and CNN extracted features / predictions. To the best of our knowledge, this is the first time all of the modalities were extracted using deep neural networks. We evaluate the performance of our approach against baselines and identify insights throughout this paper.
References in corpus (6)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Sequence to Sequence Learning with Neural Networks
- Deep Residual Learning for Image Recognition
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- R-CNNs for Pose Estimation and Action Detection