Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey

Abstract

Deep learning has made significant progress in computer vision, specifically in image classification, object detection, and semantic segmentation. The skip connection has played an essential role in the architecture of deep neural networks,enabling easier optimization through residual learning during the training stage and improving accuracy during testing. Many neural networks have inherited the idea of residual learning with skip connections for various tasks, and it has been the standard choice for designing neural networks. This survey provides a comprehensive summary and outlook on the development of skip connections in deep neural networks. The short history of skip connections is outlined, and the development of residual learning in deep neural networks is surveyed. The effectiveness of skip connections in the training and testing stages is summarized, and future directions for using skip connections in residual learning are discussed. Finally, we summarize seminal papers, source code, models, and datasets that utilize skip connections in computer vision, including image classification, object detection, semantic segmentation, and image reconstruction. We hope this survey could inspire peer researchers in the community to develop further skip connections in various forms and tasks and the theory of residual learning in deep neural networks. The project page can be found at https://github.com/apple1986/Residual_Learning_For_Images

References in corpus (30)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Deep Learning in Neural Networks: An Overview
• Automated Design of Deep Learning Methods for Biomedical Image Segmentation
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Generative Adversarial Networks: An Overview
• LLaMA: Open and Efficient Foundation Language Models
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Res2Net: A New Multi-scale Backbone Architecture
• U${}^2$-Net: Going Deeper with Nested U-Structure for Salient Object Detection
• ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data
• Diffusion Models in Vision: A Survey
• MLP-Mixer: An all-MLP Architecture for Vision
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A survey of loss functions for semantic segmentation
• Visualizing the Loss Landscape of Neural Nets
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• A survey on modern trainable activation functions
• Improved Denoising Diffusion Probabilistic Models
• Shake-Shake regularization
• A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstruction
• ShakeDrop Regularization for Deep Residual Learning
• Deformable 3D Convolution for Video Super-Resolution
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Skip Connections Eliminate Singularities
• Implicit Regularization in Deep Learning
• DarkDeblur: Learning single-shot image deblurring in low-light condition