Striving for Simplicity: The All Convolutional Net

Abstract

Most modern convolutional neural networks (CNNs) used for object recognition are built using the same principles: Alternating convolution and max-pooling layers followed by a small number of fully connected layers. We re-evaluate the state of the art for object recognition from small images with convolutional networks, questioning the necessity of different components in the pipeline. We find that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks. Following this finding -- and building on other recent work for finding simple network structures -- we propose a new architecture that consists solely of convolutional layers and yields competitive or state of the art performance on several object recognition datasets (CIFAR-10, CIFAR-100, ImageNet). To analyze the network we introduce a new variant of the "deconvolution approach" for visualizing features learned by CNNs, which can be applied to a broader range of network structures than existing approaches.

accepted to ICLR-2015 workshop track; no changes other than style

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Fractional Max-Pooling
• High-Performance Neural Networks for Visual Object Classification
• Deep Networks with Internal Selective Attention through Feedback Connections

Cited by in corpus (99)

• Neural Architecture Search with Reinforcement Learning
• Methods for Interpreting and Understanding Deep Neural Networks
• Temporal Ensembling for Semi-Supervised Learning
• A deep learning architecture for temporal sleep stage classification using multivariate and multimodal time series
• Residual Networks of Residual Networks: Multilevel Residual Networks
• Grad-CAM: Why did you say that?
• Explainable AI for Trees: From Local Explanations to Global Understanding
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Do Convolutional Neural Networks Learn Class Hierarchy?
• LipNet: End-to-End Sentence-level Lipreading
• Ensemble of Deep Convolutional Neural Networks for Automatic Pavement Crack Detection and Measurement
• Further advantages of data augmentation on convolutional neural networks
• B-CNN: Branch Convolutional Neural Network for Hierarchical Classification
• Theory and Tools for the Conversion of Analog to Spiking Convolutional Neural Networks
• Wavelet Convolutional Neural Networks for Texture Classification
• A Deep Learning Perspective on the Origin of Facial Expressions
• Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples
• Deep Koalarization: Image Colorization using CNNs and Inception-ResNet-v2
• Deep Divergence-Based Approach to Clustering
• Interleaved Group Convolutions for Deep Neural Networks
• Gradients of Counterfactuals
• Real Time Image Saliency for Black Box Classifiers
• On the Potential of Simple Framewise Approaches to Piano Transcription
• Real-time Convolutional Neural Networks for Emotion and Gender Classification
• $\mathcal{G}$-softmax: Improving Intra-class Compactness and Inter-class Separability of Features
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data
• Deep Convolutional Neural Network Design Patterns
• Separating the EoR Signal with a Convolutional Denoising Autoencoder: A Deep-learning-based Method
• Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift
• Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
• An Analysis of Pre-Training on Object Detection
• DeepMRSeg: A convolutional deep neural network for anatomy and abnormality segmentation on MR images
• Image Classification with Hierarchical Multigraph Networks
• Interpretable deep learning for nuclear deformation in heavy ion collisions
• Attention Based Glaucoma Detection: A Large-scale Database and CNN Model
• Relevance Prediction from Eye-movements Using Semi-interpretable Convolutional Neural Networks
• Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
• Combining Strategic Learning and Tactical Search in Real-Time Strategy Games
• Parle: parallelizing stochastic gradient descent
• Take it in your stride: Do we need striding in CNNs?
• Towards Score Following in Sheet Music Images
• Knowledge Projection for Deep Neural Networks
• Learning Pixel-Distribution Prior with Wider Convolution for Image Denoising
• Towards Robust, Locally Linear Deep Networks
• Information Dropout: Learning Optimal Representations Through Noisy Computation
• Optimization of Convolutional Neural Network using Microcanonical Annealing Algorithm
• Age Group and Gender Estimation in the Wild with Deep RoR Architecture
• Dropout with Expectation-linear Regularization
• DVOLVER: Efficient Pareto-Optimal Neural Network Architecture Search
• Convolutional Residual Memory Networks
• DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition
• Attention-based Wav2Text with Feature Transfer Learning
• Spatial-Winograd Pruning Enabling Sparse Winograd Convolution
• Urban morphology meets deep learning: Exploring urban forms in one million cities, town and villages across the planet
• Self-explanatory Deep Salient Object Detection
• Saliency-driven Word Alignment Interpretation for Neural Machine Translation
• Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning
• Convolutional Networks with Dense Connectivity
• Regularizing Reasons for Outfit Evaluation with Gradient Penalty
• A novel method for extracting interpretable knowledge from a spiking neural classifier with time-varying synaptic weights
• Multiscale Hierarchical Convolutional Networks
• Out-of-Distribution Detection Using Neural Rendering Generative Models
• Transfer entropy-based feedback improves performance in artificial neural networks
• Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples
• Stochastic Gradient Descent with Polyak's Learning Rate
• Improving Object Detection with Inverted Attention
• Towards Interrogating Discriminative Machine Learning Models
• Convolution Aware Initialization
• SISC: End-to-end Interpretable Discovery Radiomics-Driven Lung Cancer Prediction via Stacked Interpretable Sequencing Cells
• A Forward-Backward Approach for Visualizing Information Flow in Deep Networks
• Bayesian Conditional Generative Adverserial Networks
• Generalizing Deep Models for Overhead Image Segmentation Through Getis-Ord Gi* Pooling
• Fast Predictive Multimodal Image Registration
• Infinite Variational Autoencoder for Semi-Supervised Learning
• Channel Locality Block: A Variant of Squeeze-and-Excitation
• Context Augmentation for Convolutional Neural Networks
• Learning Less-Overlapping Representations
• Deep Visual City Recognition Visualization
• Improving training of deep neural networks via Singular Value Bounding
• Using KL-divergence to focus Deep Visual Explanation
• Visual Confusion Label Tree For Image Classification
• Fine-grained Uncertainty Modeling in Neural Networks
• Building a Regular Decision Boundary with Deep Networks
• Fixed smooth convolutional layer for avoiding checkerboard artifacts in CNNs
• Top-down Neural Attention by Excitation Backprop
• Adversarial TCAV -- Robust and Effective Interpretation of Intermediate Layers in Neural Networks
• Optimistic and Pessimistic Neural Networks for Scene and Object Recognition
• CrescendoNet: A Simple Deep Convolutional Neural Network with Ensemble Behavior
• Transparency guided ensemble convolutional neural networks for stratification of pseudoprogression and true progression of glioblastoma multiform
• Software-Defined FPGA Accelerator Design for Mobile Deep Learning Applications
• A Fully Trainable Network with RNN-based Pooling
• Learning Preference-Based Similarities from Face Images using Siamese Multi-Task CNNs
• Neuron Segmentation Using Deep Complete Bipartite Networks
• Are Saddles Good Enough for Deep Learning?
• Exploring the influence of scale on artist attribution
• An Analysis of Human-centered Geolocation
• A backward pass through a CNN using a generative model of its activations