Deep Neural Networks with Short Circuits for Improved Gradient Learning

Abstract

Deep neural networks have achieved great success both in computer vision and natural language processing tasks. However, mostly state-of-art methods highly rely on external training or computing to improve the performance. To alleviate the external reliance, we proposed a gradient enhancement approach, conducted by the short circuit neural connections, to improve the gradient learning of deep neural networks. The proposed short circuit is a unidirectional connection that single back propagates the sensitive from the deep layer to the shallows. Moreover, the short circuit formulates to be a gradient truncation of its crossing layers which can plug into the backbone deep neural networks without introducing external training parameters. Extensive experiments demonstrate deep neural networks with our short circuit gain a large margin over the baselines on both computer vision and natural language processing tasks.

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• On the difficulty of training Recurrent Neural Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Densely Connected Convolutional Networks
• Training Very Deep Networks
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Noisy Differentiable Architecture Search
• MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension