Return of the Devil in the Details: Delving Deep into Convolutional Nets

Abstract

The latest generation of Convolutional Neural Networks (CNN) have achieved impressive results in challenging benchmarks on image recognition and object detection, significantly raising the interest of the community in these methods. Nevertheless, it is still unclear how different CNN methods compare with each other and with previous state-of-the-art shallow representations such as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper conducts a rigorous evaluation of these new techniques, exploring different deep architectures and comparing them on a common ground, identifying and disclosing important implementation details. We identify several useful properties of CNN-based representations, including the fact that the dimensionality of the CNN output layer can be reduced significantly without having an adverse effect on performance. We also identify aspects of deep and shallow methods that can be successfully shared. In particular, we show that the data augmentation techniques commonly applied to CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost. Source code and models to reproduce the experiments in the paper is made publicly available.

arXiv commentPublished in proceedings of BMVC 2014

References in corpus (3)

• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Visualizing and Understanding Convolutional Networks

Cited by in corpus (178)

• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• VoxCeleb2: Deep Speaker Recognition
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• The THUMOS Challenge on Action Recognition for Videos "in the Wild"
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Fixed Point Quantization of Deep Convolutional Networks
• Transductive Multi-view Zero-Shot Learning
• Land Use Classification in Remote Sensing Images by Convolutional Neural Networks
• Smart Augmentation - Learning an Optimal Data Augmentation Strategy
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Recent Advances in Convolutional Neural Networks
• SMILES Enumeration as Data Augmentation for Neural Network Modeling of Molecules
• Modeling and Propagating CNNs in a Tree Structure for Visual Tracking
• Lung and Pancreatic Tumor Characterization in the Deep Learning Era: Novel Supervised and Unsupervised Learning Approaches
• Multi-view Convolutional Neural Networks for 3D Shape Recognition
• Context-aware Deep Feature Compression for High-speed Visual Tracking
• Deep Learning for Generic Object Detection: A Survey
• Effective Multi-Query Expansions: Collaborative Deep Networks for Robust Landmark Retrieval
• Deep Supervised Discrete Hashing
• What-and-Where to Match: Deep Spatially Multiplicative Integration Networks for Person Re-identification
• BoxCars: Improving Fine-Grained Recognition of Vehicles using 3-D Bounding Boxes in Traffic Surveillance
• BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation
• Shallow and Deep Convolutional Networks for Saliency Prediction
• Robustness of classifiers: from adversarial to random noise
• Stripe-based and Attribute-aware Network: A Two-Branch Deep Model for Vehicle Re-identification
• Self Paced Deep Learning for Weakly Supervised Object Detection
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• Do Adversarially Robust ImageNet Models Transfer Better?
• Wavelet Convolutional Neural Networks
• Fast Feature Fool: A data independent approach to universal adversarial perturbations
• What Do We Understand About Convolutional Networks?
• Wavelet Convolutional Neural Networks for Texture Classification
• Generalisation and Sharing in Triplet Convnets for Sketch based Visual Search
• Universal adversarial perturbations
• Fast Fine-grained Image Classification via Weakly Supervised Discriminative Localization
• Neural Activation Constellations: Unsupervised Part Model Discovery with Convolutional Networks
• Learning Multi-Domain Convolutional Neural Networks for Visual Tracking
• DiracNets: Training Very Deep Neural Networks Without Skip-Connections
• Do We Really Need to Collect Millions of Faces for Effective Face Recognition?
• Bilinear CNNs for Fine-grained Visual Recognition
• Improved Recurrent Neural Networks for Session-based Recommendations
• Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks
• Automatic identification of fossils and abiotic grains during carbonate microfacies analysis using deep convolutional neural networks
• Deep convolutional filter banks for texture recognition and segmentation
• RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints
• Compact Bilinear Pooling
• Prospects for Theranostics in Neurosurgical Imaging: Empowering Confocal Laser Endomicroscopy Diagnostics via Deep Learning
• Understanding trained CNNs by indexing neuron selectivity
• Feedforward semantic segmentation with zoom-out features
• Delta Networks for Optimized Recurrent Network Computation
• GIFT: A Real-time and Scalable 3D Shape Search Engine
• Sketch-a-Net that Beats Humans
• Depth from a Single Image by Harmonizing Overcomplete Local Network Predictions
• Stacked Quantizers for Compositional Vector Compression
• Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes
• WarpNet: Weakly Supervised Matching for Single-view Reconstruction
• Fisher Kernel for Deep Neural Activations
• Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs
• Is Second-order Information Helpful for Large-scale Visual Recognition?
• Deep Convolutional Features for Image Based Retrieval and Scene Categorization
• Accelerating Very Deep Convolutional Networks for Classification and Detection
• Improved Bilinear Pooling with CNNs
• A Discriminative CNN Video Representation for Event Detection
• Simple Image Description Generator via a Linear Phrase-Based Approach
• Multiple Instance Detection Network with Online Instance Classifier Refinement
• Semi-supervised Vocabulary-informed Learning
• Convolutional Neural Networks at Constrained Time Cost
• Going Deeper into First-Person Activity Recognition
• A Survey on Deep Learning Toolkits and Libraries for Intelligent User Interfaces
• When Unsupervised Domain Adaptation Meets Tensor Representations
• Texture Synthesis Through Convolutional Neural Networks and Spectrum Constraints
• Soft Proposal Networks for Weakly Supervised Object Localization
• Video-based Human Action Recognition using Deep Learning: A Review
• What's Mine is Yours: Pretrained CNNs for Limited Training Sonar ATR
• Cross Modal Distillation for Supervision Transfer
• Actionness Estimation Using Hybrid Fully Convolutional Networks
• Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization
• Deep Learning for Object Saliency Detection and Image Segmentation
• Not Afraid of the Dark: NIR-VIS Face Recognition via Cross-spectral Hallucination and Low-rank Embedding
• Optimizing the Trade-off between Single-Stage and Two-Stage Object Detectors using Image Difficulty Prediction
• HashGAN:Attention-aware Deep Adversarial Hashing for Cross Modal Retrieval
• Visualizing and Understanding Deep Texture Representations
• Privacy-Preserving Deep Inference for Rich User Data on The Cloud
• FashionNet: Personalized Outfit Recommendation with Deep Neural Network
• Automatic Classification of Bright Retinal Lesions via Deep Network Features
• Convolutional Neural Network-Based Image Representation for Visual Loop Closure Detection
• Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
• Deep Convolutional Neural Fields for Depth Estimation from a Single Image
• ScreenAvoider: Protecting Computer Screens from Ubiquitous Cameras
• Temporal Dynamic Graph LSTM for Action-driven Video Object Detection
• Digging Deep into the layers of CNNs: In Search of How CNNs Achieve View Invariance
• Deep Representation of Facial Geometric and Photometric Attributes for Automatic 3D Facial Expression Recognition
• AttentionNet: Aggregating Weak Directions for Accurate Object Detection
• Zero-Shot Domain Adaptation via Kernel Regression on the Grassmannian
• Unsupervised Category Discovery via Looped Deep Pseudo-Task Optimization Using a Large Scale Radiology Image Database
• Convolutional Models for Joint Object Categorization and Pose Estimation
• A Jointly Learned Deep Architecture for Facial Attribute Analysis and Face Detection in the Wild
• Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)
• Cultural Event Recognition with Visual ConvNets and Temporal Models
• Holistic Interstitial Lung Disease Detection using Deep Convolutional Neural Networks: Multi-label Learning and Unordered Pooling
• Recurrent Filter Learning for Visual Tracking
• Assessing fish abundance from underwater video using deep neural networks
• Differential Angular Imaging for Material Recognition
• ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks using Angle Sensitive Pixels
• Simple and Efficient Learning using Privileged Information
• Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
• Is Image Super-resolution Helpful for Other Vision Tasks?
• Fine-tuning deep CNN models on specific MS COCO categories
• Correlation Tracking via Joint Discrimination and Reliability Learning
• Multi-Modal Music Information Retrieval: Augmenting Audio-Analysis with Visual Computing for Improved Music Video Analysis
• Integrated perception with recurrent multi-task neural networks
• Multi-hypothesis contextual modeling for semantic segmentation
• Object-Scene Convolutional Neural Networks for Event Recognition in Images
• Unsupervised Joint Mining of Deep Features and Image Labels for Large-scale Radiology Image Categorization and Scene Recognition
• MIHash: Online Hashing with Mutual Information
• ProNet: Learning to Propose Object-specific Boxes for Cascaded Neural Networks
• Action Recognition for Depth Video using Multi-view Dynamic Images
• Asymmetric Deep Supervised Hashing
• When Naïve Bayes Nearest Neighbours Meet Convolutional Neural Networks
• Visual Summary of Egocentric Photostreams by Representative Keyframes
• LEWIS: Latent Embeddings for Word Images and their Semantics
• Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification
• Where to Focus: Deep Attention-based Spatially Recurrent Bilinear Networks for Fine-Grained Visual Recognition
• Domain-Size Pooling in Local Descriptors: DSP-SIFT
• A Large-scale Attribute Dataset for Zero-shot Learning
• How hard can it be? Estimating the difficulty of visual search in an image
• Exploit Bounding Box Annotations for Multi-label Object Recognition
• Accurate Facial Parts Localization and Deep Learning for 3D Facial Expression Recognition
• Bi-class classification of humpback whale sound units against complex background noise with Deep Convolution Neural Network
• On monitoring development indicators using high resolution satellite images
• Transfer Learning for Action Unit Recognition
• Real-Time Rotation-Invariant Face Detection with Progressive Calibration Networks
• P-CNN: Pose-based CNN Features for Action Recognition
• A New Target-specific Object Proposal Generation Method for Visual Tracking
• Learning Discriminative 3D Shape Representations by View Discerning Networks
• Exploiting Multi-modal Curriculum in Noisy Web Data for Large-scale Concept Learning
• Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition
• Material Classification in the Wild: Do Synthesized Training Data Generalise Better than Real-World Training Data?
• Fast Loop Closure Detection via Binary Content
• Learning Deep Representations for Scene Labeling with Semantic Context Guided Supervision
• Towards CNN Map Compression for camera relocalisation
• CNN Feature boosted SeqSLAM for Real-Time Loop Closure Detection
• No Spare Parts: Sharing Part Detectors for Image Categorization
• Geometric Convolutional Neural Network for Analyzing Surface-Based Neuroimaging Data
• Semantic tracking: Single-target tracking with inter-supervised convolutional networks
• Differential Geometric Retrieval of Deep Features
• Object Detection with Mask-based Feature Encoding
• From Images to 3D Shape Attributes
• Efficient On-the-fly Category Retrieval using ConvNets and GPUs
• Detecting Dominant Vanishing Points in Natural Scenes with Application to Composition-Sensitive Image Retrieval
• Learnable Histogram: Statistical Context Features for Deep Neural Networks
• Improving Human Action Recognition by Non-action Classification
• Design of Kernels in Convolutional Neural Networks for Image Classification
• Make Your Bone Great Again : A study on Osteoporosis Classification
• Real-time Online Action Detection Forests using Spatio-temporal Contexts
• Learning to Select Pre-Trained Deep Representations with Bayesian Evidence Framework
• Higher-order Pooling of CNN Features via Kernel Linearization for Action Recognition
• Face-space Action Recognition by Face-Object Interactions
• AFFACT - Alignment-Free Facial Attribute Classification Technique
• Detecting abnormal events in video using Narrowed Normality Clusters
• Dynamic texture and scene classification by transferring deep image features
• Do Deep Neural Networks Model Nonlinear Compositionality in the Neural Representation of Human-Object Interactions?
• Deep Collaborative Learning for Visual Recognition
• Learning Robust Hash Codes for Multiple Instance Image Retrieval
• The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
• Compositional Model based Fisher Vector Coding for Image Classification
• Visual Search at Pinterest
• Convolutional Architecture Exploration for Action Recognition and Image Classification
• A novel learning-based frame pooling method for Event Detection
• MV-C3D: A Spatial Correlated Multi-View 3D Convolutional Neural Networks
• Weakly supervised segment annotation via expectation kernel density estimation
• Deep Tensor Encoding
• An Analysis of Human-centered Geolocation
• Gram Regularization for Multi-view 3D Shape Retrieval
• Hashing with Binary Matrix Pursuit
• Learning to Compose with Professional Photographs on the Web