Identity Mappings in Deep Residual Networks

Abstract

Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers

ECCV 2016 camera-ready

References in corpus (4)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Striving for Simplicity: The All Convolutional Net
• Fractional Max-Pooling

Cited by in corpus (108)

• Neural Architecture Search with Reinforcement Learning
• ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data
• SGDR: Stochastic Gradient Descent with Warm Restarts
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• Trained Ternary Quantization
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Human pose estimation via Convolutional Part Heatmap Regression
• Training Deep Nets with Sublinear Memory Cost
• Generating Long Sequences with Sparse Transformers
• Residual Networks of Residual Networks: Multilevel Residual Networks
• Residual Attention Network for Image Classification
• Deep Networks with Stochastic Depth
• A2D2: Audi Autonomous Driving Dataset
• Revisiting ResNets: Improved Training and Scaling Strategies
• CMU DeepLens: Deep Learning For Automatic Image-based Galaxy-Galaxy Strong Lens Finding
• Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
• Improving Variational Inference with Inverse Autoregressive Flow
• Deep Complex Networks
• Bridging Category-level and Instance-level Semantic Image Segmentation
• The Strong Gravitational Lens Finding Challenge
• CNN-based Segmentation of Medical Imaging Data
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• Context Encoding for Semantic Segmentation
• Deep Generative Adversarial Networks for Compressed Sensing Automates MRI
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• VoxResNet: Deep Voxelwise Residual Networks for Volumetric Brain Segmentation
• Finding Strong Gravitational Lenses in the DESI DECam Legacy Survey
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• On the Origin of Deep Learning
• DIVA: Domain Invariant Variational Autoencoders
• High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks
• PVANet: Lightweight Deep Neural Networks for Real-time Object Detection
• Embedded Binarized Neural Networks
• RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation
• Deep Hyperspherical Learning
• Receptive Field Regularization Techniques for Audio Classification and Tagging with Deep Convolutional Neural Networks
• Detecting Adversarial Examples by Input Transformations, Defense Perturbations, and Voting
• A Survey on Deep Learning Methods for Robot Vision
• HOLISMOKES. VIII. High-redshift, strong-lens search in the Hyper Suprime-Cam Subaru Strategic Program
• Deep Convolutional Neural Network Design Patterns
• Neutrino interaction classification with a convolutional neural network in the DUNE far detector
• Multi-task, multi-label and multi-domain learning with residual convolutional networks for emotion recognition
• Flood-Filling Networks
• Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
• Multi-Objective De Novo Drug Design with Conditional Graph Generative Model
• A deep learning view of the census of galaxy clusters in IllustrisTNG
• Large-Scale Visual Active Learning with Deep Probabilistic Ensembles
• Explaining deep learning of galaxy morphology with saliency mapping
• Quark-Gluon Jet Discrimination Using Convolutional Neural Networks
• Deep Learning for Identifying Iran's Cultural Heritage Buildings in Need of Conservation Using Image Classification and Grad-CAM
• On the Validity of Bayesian Neural Networks for Uncertainty Estimation
• Fixing the train-test resolution discrepancy
• NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data
• Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification
• Prospects for future studies using deep imaging: Analysis of individual Galactic cirrus filaments
• Finding Better Topologies for Deep Convolutional Neural Networks by Evolution
• Deep Illumination: Approximating Dynamic Global Illumination with Generative Adversarial Network
• Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuning
• Modelling Force-Free Neutron Star Magnetospheres using Physics-Informed Neural Networks
• Deep Recurrent Convolutional Neural Network: Improving Performance For Speech Recognition
• Detection of Strongly Lensed Arcs in Galaxy Clusters with Transformers
• Inverse Compositional Spatial Transformer Networks
• Age Group and Gender Estimation in the Wild with Deep RoR Architecture
• Single Image Super-resolution via a Lightweight Residual Convolutional Neural Network
• Bounded Residual Gradient Networks (BReG-Net) for Facial Affect Computing
• Modularized Morphing of Neural Networks
• Improved Stereo Matching with Constant Highway Networks and Reflective Confidence Learning
• Convolutional Residual Memory Networks
• Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Data
• Seismic-Net: A Deep Densely Connected Neural Network to Detect Seismic Events
• How deep learning works --The geometry of deep learning
• Face Detection with Feature Pyramids and Landmarks
• Visual Compiler: Synthesizing a Scene-Specific Pedestrian Detector and Pose Estimator
• Deep TEN: Texture Encoding Network
• Compressed Sensing: From Research to Clinical Practice with Data-Driven Learning
• The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
• Deep Cross Residual Learning for Multitask Visual Recognition
• Real-time Human Pose Estimation from Video with Convolutional Neural Networks
• Beyond Fine Tuning: A Modular Approach to Learning on Small Data
• A Unified Framework for Tumor Proliferation Score Prediction in Breast Histopathology
• The Loss Surface of Residual Networks: Ensembles and the Role of Batch Normalization
• One-to-Many Network for Visually Pleasing Compression Artifacts Reduction
• Adversarial Image Alignment and Interpolation
• Skeleton based Activity Recognition by Fusing Part-wise Spatio-temporal and Attention Driven Residues
• Visual Fashion-Product Search at SK Planet
• Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification
• CNN based texture synthesize with Semantic segment
• Looking for change? Roll the Dice and demand Attention
• Refined Deep Neural Network and U-Net for Polyps Segmentation
• Gravilon: Applications of a New Gradient Descent Method to Machine Learning
• Stacked Approximated Regression Machine: A Simple Deep Learning Approach
• Tuning IR-cut Filter for Illumination-aware Spectral Reconstruction from RGB
• Prune2Edge: A Multi-Phase Pruning Pipelines to Deep Ensemble Learning in IIoT
• Neuron Segmentation Using Deep Complete Bipartite Networks
• Smart Device based Initial Movement Detection of Cyclists using Convolutional Neuronal Networks
• Weight mechanism: adding a constant in concatenation of series connect
• Robust Temporal Ensembling for Learning with Noisy Labels
• Occlusion-guided compact template learning for ensemble deep network-based pose-invariant face recognition
• Local Contrast Learning
• Image Classification Algorithm for Determining the Light Curve Morphologies of ASAS-SN Eclipsing Binaries
• Self-Refining Deep Symmetry Enhanced Network for Rain Removal
• Deep Competitive Pathway Networks
• Mixing between the Cross Entropy and the Expectation Loss Terms
• Simple2Complex: Global Optimization by Gradient Descent
• Safe Augmentation: Learning Task-Specific Transformations from Data
• CloudifierNet -- Deep Vision Models for Artificial Image Processing