Generalization in Machine Learning via Analytical Learning Theory
arXiv:1802.07426
Abstract
This paper introduces a novel measure-theoretic theory for machine learning that does not require statistical assumptions. Based on this theory, a new regularization method in deep learning is derived and shown to outperform previous methods in CIFAR-10, CIFAR-100, and SVHN. Moreover, the proposed theory provides a theoretical basis for a family of practically successful regularization methods in deep learning. We discuss several consequences of our results on one-shot learning, representation learning, deep learning, and curriculum learning. Unlike statistical learning theory, the proposed learning theory analyzes each problem instance individually via measure theory, rather than a set of problem instances via statistics. As a result, it provides different types of results and insights when compared to statistical learning theory.
References in corpus (15)
- Improved Regularization of Convolutional Neural Networks with Cutout
- Wide Residual Networks
- The Loss Surfaces of Multilayer Networks
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Deep Learning Scaling is Predictable, Empirically
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- A Closer Look at Memorization in Deep Networks
- Distributional Smoothing with Virtual Adversarial Training
- Better Mixing via Deep Representations
- Sharp Minima Can Generalize For Deep Nets
- Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
- Bayesian Optimization with Exponential Convergence
- Exponentially vanishing sub-optimal local minima in multilayer neural networks
- Theory of Deep Learning III: explaining the non-overfitting puzzle
- Ensemble Robustness and Generalization of Stochastic Deep Learning Algorithms
Cited by in corpus (6)
- Generalization in Deep Learning
- On Feature Normalization and Data Augmentation
- Deep Active Learning with Augmentation-based Consistency Estimation
- Semi-Supervised Learning by Disentangling and Self-Ensembling Over Stochastic Latent Space
- Improving Generalization of Sequence Encoder-Decoder Networks for Inverse Imaging of Cardiac Transmembrane Potential
- Improving Generalization of Deep Networks for Inverse Reconstruction of Image Sequences