Deeply-Supervised Nets

Abstract

Our proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent. We make an attempt to boost the classification performance by studying a new formulation in deep networks. Three aspects in convolutional neural networks (CNN) style architectures are being looked at: (1) transparency of the intermediate layers to the overall classification; (2) discriminativeness and robustness of learned features, especially in the early layers; (3) effectiveness in training due to the presence of the exploding and vanishing gradients. We introduce "companion objective" to the individual hidden layers, in addition to the overall objective at the output layer (a different strategy to layer-wise pre-training). We extend techniques from stochastic gradient methods to analyze our algorithm. The advantage of our method is evident and our experimental result on benchmark datasets shows significant performance gain over existing methods (e.g. all state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and SVHN).

Patent disclosure, UCSD Docket No. SD2014-313, filed on May 22, 2014

References in corpus (7)

• Improving neural networks by preventing co-adaptation of feature detectors
• On the difficulty of training Recurrent Neural Networks
• Deep Learning using Linear Support Vector Machines
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Visualizing and Understanding Convolutional Networks
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima

Cited by in corpus (137)

• Deep Residual Learning for Image Recognition
• Striving for Simplicity: The All Convolutional Net
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• FitNets: Hints for Thin Deep Nets
• Wide Residual Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Training Very Deep Networks
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Medical Image Segmentation Using Deep Learning: A Survey
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• Density estimation using Real NVP
• DASNet: Dual attentive fully convolutional siamese networks for change detection of high resolution satellite images
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Learning to Diagnose with LSTM Recurrent Neural Networks
• Data Augmentation using Random Image Cropping and Patching for Deep CNNs
• Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration
• Residual Networks of Residual Networks: Multilevel Residual Networks
• Learning Activation Functions to Improve Deep Neural Networks
• Deep Networks with Stochastic Depth
• Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
• Automatic calcium scoring in low-dose chest CT using deep neural networks with dilated convolutions
• Direction-aware Spatial Context Features for Shadow Detection and Removal
• Learning a Probabilistic Model for Diffeomorphic Registration
• Stacked What-Where Auto-encoders
• Deep Pictorial Gaze Estimation
• 3D Segmentation with Exponential Logarithmic Loss for Highly Unbalanced Object Sizes
• FAMED-Net: A Fast and Accurate Multi-scale End-to-end Dehazing Network
• HED-UNet: Combined Segmentation and Edge Detection for Monitoring the Antarctic Coastline
• Gland Instance Segmentation Using Deep Multichannel Neural Networks
• Spectral Representations for Convolutional Neural Networks
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
• Training Deeper Convolutional Networks with Deep Supervision
• Deep Residual Correction Network for Partial Domain Adaptation
• Discrimination-aware Network Pruning for Deep Model Compression
• CNN-based Segmentation of Medical Imaging Data
• DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation
• Edge Preserving and Multi-Scale Contextual Neural Network for Salient Object Detection
• Deep Networks for Image Super-Resolution with Sparse Prior
• Deep Residual Networks with Exponential Linear Unit
• A Deep Learning Reconstruction Framework for Differential Phase-Contrast Computed Tomography with Incomplete Data
• 3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes
• APAC: Augmented PAttern Classification with Neural Networks
• DeepSkeleton: Learning Multi-task Scale-associated Deep Side Outputs for Object Skeleton Extraction in Natural Images
• Local-Global Context Aware Transformer for Language-Guided Video Segmentation
• Optimized Deep Encoder-Decoder Methods for Crack Segmentation
• Locally-Supervised Deep Hybrid Model for Scene Recognition
• Real-time High-Resolution Neural Network with Semantic Guidance for Crack Segmentation
• LE-UDA: Label-efficient unsupervised domain adaptation for medical image segmentation
• Fast, simple and accurate handwritten digit classification by training shallow neural network classifiers with the 'extreme learning machine' algorithm
• DiracNets: Training Very Deep Neural Networks Without Skip-Connections
• Building medical image classifiers with very limited data using segmentation networks
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• A Neural Network based Shock Detection and Localization Approach for Discontinuous Galerkin Methods
• Variational Temporal Deep Generative Model for Radar HRRP Target Recognition
• Domain adaptation based self-correction model for COVID-19 infection segmentation in CT images
• Feature Pyramid and Hierarchical Boosting Network for Pavement Crack Detection
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• Convolutional Neural Networks with Gated Recurrent Connections
• EDITH :ECG biometrics aided by Deep learning for reliable Individual auTHentication
• $\mathcal{G}$-softmax: Improving Intra-class Compactness and Inter-class Separability of Features
• Generalized Wasserstein Dice Score, Distributionally Robust Deep Learning, and Ranger for brain tumor segmentation: BraTS 2020 challenge
• Imbalanced Learning-based Automatic SAR Images Change Detection by Morphologically Supervised PCA-Net
• Early Exit or Not: Resource-Efficient Blind Quality Enhancement for Compressed Images
• Deeply-Recursive Convolutional Network for Image Super-Resolution
• AugFPN: Improving Multi-scale Feature Learning for Object Detection
• Alzheimer's Disease Diagnostics by a Deeply Supervised Adaptable 3D Convolutional Network
• Higher Order Recurrent Neural Networks
• HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition
• Deep Ice Layer Tracking and Thickness Estimation using Fully Convolutional Networks
• Competitive Multi-scale Convolution
• Embedded deep learning in ophthalmology: Making ophthalmic imaging smarter
• Triangle-Net: Towards Robustness in Point Cloud Learning
• A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
• Towards a Visual Turing Challenge
• A Deep Learning Approach for Pose Estimation from Volumetric OCT Data
• Deeply Self-Supervised Contour Embedded Neural Network Applied to Liver Segmentation
• Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
• Doubly Convolutional Neural Networks
• A Novel Weight-Shared Multi-Stage CNN for Scale Robustness
• Online Semi-Supervised Learning with Deep Hybrid Boltzmann Machines and Denoising Autoencoders
• Colorful Image Colorization
• Spatial Aggregation of Holistically-Nested Networks for Automated Pancreas Segmentation
• Boundary-semantic collaborative guidance network with dual-stream feedback mechanism for salient object detection in optical remote sensing imagery
• Equalization Loss for Long-Tailed Object Recognition
• Uncertainty Injection: A Deep Learning Method for Robust Optimization
• Adaptive Regularization of Labels
• Learning Kernel for Conditional Moment-Matching Discrepancy-based Image Classification
• Conducting Credit Assignment by Aligning Local Representations
• CrackCLF: Automatic Pavement Crack Detection based on Closed-Loop Feedback
• Spatial Aggregation of Holistically-Nested Convolutional Neural Networks for Automated Pancreas Localization and Segmentation
• On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case
• An Algorithm for Learning Shape and Appearance Models without Annotations
• Learning to Adapt by Minimizing Discrepancy
• Guided Variational Autoencoder for Disentanglement Learning
• Globally Tuned Cascade Pose Regression via Back Propagation with Application in 2D Face Pose Estimation and Heart Segmentation in 3D CT Images
• Basic Level Categorization Facilitates Visual Object Recognition
• Learning A Task-Specific Deep Architecture For Clustering
• Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks
• Semantic keypoint-based pose estimation from single RGB frames
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• Deep Roto-Translation Scattering for Object Classification
• Distributionally Robust Deep Learning using Hardness Weighted Sampling
• On the Depth of Deep Neural Networks: A Theoretical View
• A Multiscale Patch Based Convolutional Network for Brain Tumor Segmentation
• Training Progressively Binarizing Deep Networks Using FPGAs
• A 1Mb mixed-precision quantized encoder for image classification and patch-based compression
• Cascaded Subpatch Networks for Effective CNNs
• Invariant backpropagation: how to train a transformation-invariant neural network
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Collaborative Layer-wise Discriminative Learning in Deep Neural Networks
• Multi-output Deep-Supervised Classifier Chains for Plant Pathology
• Convolutional Oriented Boundaries: From Image Segmentation to High-Level Tasks
• Dual Encoder Fusion U-Net (DEFU-Net) for Cross-manufacturer Chest X-ray Segmentation
• FNOSeg3D: Resolution-Robust 3D Image Segmentation with Fourier Neural Operator
• Contextual Classification Using Self-Supervised Auxiliary Models for Deep Neural Networks
• Automatic Moth Detection from Trap Images for Pest Management
• PASCAL Boundaries: A Class-Agnostic Semantic Boundary Dataset
• Unsupervised Feature Learning with C-SVDDNet
• HartleyMHA: Self-Attention in Frequency Domain for Resolution-Robust and Parameter-Efficient 3D Image Segmentation
• DeMeshNet: Blind Face Inpainting for Deep MeshFace Verification
• Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning
• Similarity-based Text Recognition by Deeply Supervised Siamese Network
• Exploiting Layerwise Convexity of Rectifier Networks with Sign Constrained Weights
• Self-supervised Lesion Change Detection and Localisation in Longitudinal Multiple Sclerosis Brain Imaging
• Deep Haar Scattering Networks
• Comb Convolution for Efficient Convolutional Architecture
• Analysis of Macula on Color Fundus Images Using Heightmap Reconstruction Through Deep Learning
• A Distributed Deep Representation Learning Model for Big Image Data Classification
• Sparsely Activated Networks: A new method for decomposing and compressing data
• Deep Learning-Based Image Kernel for Inductive Transfer
• The Compressed Model of Residual CNDS
• A Closed-Form Learned Pooling for Deep Classification Networks
• Residual-CNDS for Grand Challenge Scene Dataset
• Rethinking Fully Convolutional Networks for the Analysis of Photoluminescence Wafer Images