Deep Quantization: Encoding Convolutional Activations with Deep Generative Model

Abstract

Deep convolutional neural networks (CNNs) have proven highly effective for visual recognition, where learning a universal representation from activations of convolutional layer plays a fundamental problem. In this paper, we present Fisher Vector encoding with Variational Auto-Encoder (FV-VAE), a novel deep architecture that quantizes the local activations of convolutional layer in a deep generative model, by training them in an end-to-end manner. To incorporate FV encoding strategy into deep generative models, we introduce Variational Auto-Encoder model, which steers a variational inference and learning in a neural network which can be straightforwardly optimized using standard stochastic gradient method. Different from the FV characterized by conventional generative models (e.g., Gaussian Mixture Model) which parsimoniously fit a discrete mixture model to data distribution, the proposed FV-VAE is more flexible to represent the natural property of data for better generalization. Extensive experiments are conducted on three public datasets, i.e., UCF101, ActivityNet, and CUB-200-2011 in the context of video action recognition and fine-grained image classification, respectively. Superior results are reported when compared to state-of-the-art representations. Most remarkably, our proposed FV-VAE achieves to-date the best published accuracy of 94.2% on UCF101.

References in corpus (22)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• ADADELTA: An Adaptive Learning Rate Method
• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Going Deeper with Convolutions
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Fully Convolutional Networks for Semantic Segmentation
• Bird Species Categorization Using Pose Normalized Deep Convolutional Nets
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Action Recognition using Visual Attention
• Composing graphical models with neural networks for structured representations and fast inference
• Sequence to Sequence -- Video to Text
• Show and Tell: A Neural Image Caption Generator
• Learning Spatiotemporal Features with 3D Convolutional Networks
• Long-term Temporal Convolutions for Action Recognition
• Random Feature Maps for Dot Product Kernels
• Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks
• Compact Bilinear Pooling
• A Discriminative CNN Video Representation for Event Detection

Cited by in corpus (4)

• Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
• End-to-end Video-level Representation Learning for Action Recognition
• Predictive Learning: Using Future Representation Learning Variantial Autoencoder for Human Action Prediction
• Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification