Nonlinear Collaborative Scheme for Deep Neural Networks

Abstract

Conventional research attributes the improvements of generalization ability of deep neural networks either to powerful optimizers or the new network design. Different from them, in this paper, we aim to link the generalization ability of a deep network to optimizing a new objective function. To this end, we propose a \textit{nonlinear collaborative scheme} for deep network training, with the key technique as combining different loss functions in a nonlinear manner. We find that after adaptively tuning the weights of different loss functions, the proposed objective function can efficiently guide the optimization process. What is more, we demonstrate that, from the mathematical perspective, the nonlinear collaborative scheme can lead to (i) smaller KL divergence with respect to optimal solutions; (ii) data-driven stochastic gradient descent; (iii) tighter PAC-Bayes bound. We also prove that its advantage can be strengthened by nonlinearity increasing. To some extent, we bridge the gap between learning (i.e., minimizing the new objective function) and generalization (i.e., minimizing a PAC-Bayes bound) in the new scheme. We also interpret our findings through the experiments on Residual Networks and DenseNet, showing that our new scheme performs superior to single-loss and multi-loss schemes no matter with randomization or not.

arXiv comment11 pages, 3 figures (20 subfigures), prepared to submit to IEEE Trans. on Neural Networks and Learning Systems

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Understanding deep learning requires rethinking generalization
• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• Domain Separation Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
• The Microsoft 2016 Conversational Speech Recognition System
• Diagnosis of diabetes using classification mining techniques
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• On Loss Functions for Deep Neural Networks in Classification
• Training behavior of deep neural network in frequency domain
• Person Re-Identification by Deep Joint Learning of Multi-Loss Classification
• Understanding training and generalization in deep learning by Fourier analysis
• Face Detection using Deep Learning: An Improved Faster RCNN Approach
• Nonlinear Acceleration of Stochastic Algorithms

Cited by in corpus (1)

• Training behavior of deep neural network in frequency domain