DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks

Abstract

This paper introduces a visual sentiment concept classification method based on deep convolutional neural networks (CNNs). The visual sentiment concepts are adjective noun pairs (ANPs) automatically discovered from the tags of web photos, and can be utilized as effective statistical cues for detecting emotions depicted in the images. Nearly one million Flickr images tagged with these ANPs are downloaded to train the classifiers of the concepts. We adopt the popular model of deep convolutional neural networks which recently shows great performance improvement on classifying large-scale web-based image dataset such as ImageNet. Our deep CNNs model is trained based on Caffe, a newly developed deep learning framework. To deal with the biased training data which only contains images with strong sentiment and to prevent overfitting, we initialize the model with the model weights trained from ImageNet. Performance evaluation shows the newly trained deep CNNs model SentiBank 2.0 (or called DeepSentiBank) is significantly improved in both annotation accuracy and retrieval performance, compared to its predecessors which mainly use binary SVM classification models.

7 pages, 4 figures

Cited by in corpus (19)

• Context Based Emotion Recognition using EMOTIC Dataset
• Visual Affect Around the World: A Large-scale Multilingual Visual Sentiment Ontology
• Survey on Visual Sentiment Analysis
• Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion
• SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network
• Seeking Subjectivity in Visual Emotion Distribution Learning
• OutdoorSent: Sentiment Analysis of Urban Outdoor Images by Using Semantic and Deep Features
• Mapping Images to Sentiment Adjective Noun Pairs with Factorized Neural Nets
• More cat than cute? Interpretable Prediction of Adjective-Noun Pairs
• Deep Cross Residual Learning for Multitask Visual Recognition
• Affective Image Content Analysis: Two Decades Review and New Perspectives
• Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Space
• Going Deeper for Multilingual Visual Sentiment Detection
• Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias
• A Multi-task Neural Approach for Emotion Attribution, Classification and Summarization
• A Blast From the Past: Personalizing Predictions of Video-Induced Emotions using Personal Memories as Context
• An Overview of Computational Approaches for Interpretation Analysis
• Sentiment Analysis from Images of Natural Disasters
• Learning to Aggregate and Refine Noisy Labels for Visual Sentiment Analysis