Statistical Mechanics of Nonlinear On-line Learning for Ensemble Teachers
arXiv:0705.2318 · doi:10.1143/JPSJ.76.114001
Abstract
We analyze the generalization performance of a student in a model composed of nonlinear perceptrons: a true teacher, ensemble teachers, and the student. We calculate the generalization error of the student analytically or numerically using statistical mechanics in the framework of on-line learning. We treat two well-known learning rules: Hebbian learning and perceptron learning. As a result, it is proven that the nonlinear model shows qualitatively different behaviors from the linear model. Moreover, it is clarified that Hebbian learning and perceptron learning show qualitatively different behaviors from each other. In Hebbian learning, we can analytically obtain the solutions. In this case, the generalization error monotonically decreases. The steady value of the generalization error is independent of the learning rate. The larger the number of teachers is and the more variety the ensemble teachers have, the smaller the generalization error is. In perceptron learning, we have to numerically obtain the solutions. In this case, the dynamical behaviors of the generalization error are non-monotonic. The smaller the learning rate is, the larger the number of teachers is; and the more variety the ensemble teachers have, the smaller the minimum value of the generalization error is.
13 pages, 9 figures
References in corpus (6)
- Analysis of ensemble learning using simple perceptrons based on online learning theory
- Online Learning with Ensembles
- Ensemble learning of linear perceptron; Online learning theory
- Statistical Mechanics of Online Learning for Ensemble Teachers
- Analysis of on-line learning when a moving teacher goes around a true teacher
- Statistical Mechanics of On-line Learning when a Moving Teacher Goes around an Unlearnable True Teacher