Modelling Temporal Information Using Discrete Fourier Transform for Recognizing Emotions in User-generated Videos

Abstract

With the widespread of user-generated Internet videos, emotion recognition in those videos attracts increasing research efforts. However, most existing works are based on framelevel visual features and/or audio features, which might fail to model the temporal information, e.g. characteristics accumulated along time. In order to capture video temporal information, in this paper, we propose to analyse features in frequency domain transformed by discrete Fourier transform (DFT features). Frame-level features are firstly extract by a pre-trained deep convolutional neural network (CNN). Then, time domain features are transferred and interpolated into DFT features. CNN and DFT features are further encoded and fused for emotion classification. By this way, static image features extracted from a pre-trained deep CNN and temporal information represented by DFT features are jointly considered for video emotion recognition. Experimental results demonstrate that combining DFT features can effectively capture temporal information and therefore improve emotion recognition performance. Our approach has achieved a state-of-the-art performance on the largest video emotion dataset (VideoEmotion-8 dataset), improving accuracy from 51.1% to 62.6%.

5 pages. arXiv admin note: substantial text overlap with arXiv:1603.06182

References in corpus (6)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Caffe: Convolutional Architecture for Fast Feature Embedding
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• A Discriminative CNN Video Representation for Event Detection
• Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification
• Temporal Extension of Scale Pyramid and Spatial Pyramid Matching for Action Recognition