Functional Gradient Boosting based on Residual Network Perception
arXiv:1802.09031
Abstract
Residual Networks (ResNets) have become state-of-the-art models in deep learning and several theoretical studies have been devoted to understanding why ResNet works so well. One attractive viewpoint on ResNet is that it is optimizing the risk in a functional space by combining an ensemble of effective features. In this paper, we adopt this viewpoint to construct a new gradient boosting method, which is known to be very powerful in data analysis. To do so, we formalize the gradient boosting perspective of ResNet mathematically using the notion of functional gradients and propose a new method called ResFGB for classification tasks by leveraging ResNet perception. Two types of generalization guarantees are provided from the optimization perspective: one is the margin bound and the other is the expected risk bound by the sample-splitting technique. Experimental results show superior performance of the proposed method over state-of-the-art methods such as LightGBM.
arXiv comment22 pages, 1 figure, 1 table. An extended version of ICML 2018 paper
References in corpus (14)
- Densely Connected Convolutional Networks
- Wide Residual Networks
- Residual Networks Behave Like Ensembles of Relatively Shallow Networks
- Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
- Norm-Based Capacity Control in Neural Networks
- Residual Connections Encourage Iterative Inference
- Reversible Architectures for Arbitrarily Deep Residual Neural Networks
- Multi-level Residual Networks from Dynamical Systems View
- Learning Deep ResNet Blocks Sequentially using Boosting Theory
- Stochastic Particle Gradient Descent for Infinite Ensembles
- Learning across scales - A multiscale method for Convolution Neural Networks
- Lecture Notes on Gradient Flows and Optimal Transport
- The Loss Surface of Residual Networks: Ensembles and the Role of Batch Normalization
- Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models