Neural Complexity Measures
arXiv:2008.02953
Abstract
While various complexity measures for deep neural networks exist, specifying an appropriate measure capable of predicting and explaining generalization in deep networks has proven challenging. We propose Neural Complexity (NC), a meta-learning framework for predicting generalization. Our model learns a scalar complexity measure through interactions with many heterogeneous tasks in a data-driven way. The trained NC model can be added to the standard training loss to regularize any task learner in a standard supervised learning scenario. We contrast NC's approach against existing manually-designed complexity measures and other meta-learning models, and we validate NC's performance on multiple regression and classification tasks
arXiv commentPublished in Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS 2020) Code is available at https://github.com/yoonholee/neural-complexity
References in corpus (19)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Prototypical Networks for Few-shot Learning
- mixup: Beyond Empirical Risk Minimization
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Bayesian Model-Agnostic Meta-Learning
- Fantastic Generalization Measures and Where to Find Them
- Predicting the Generalization Gap in Deep Networks with Margin Distributions
- Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
- Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
- Norm-Based Capacity Control in Neural Networks
- Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace
- Generalization in Deep Networks: The Role of Distance from Initialization
- Understanding Short-Horizon Bias in Stochastic Meta-Optimization
- Predicting Neural Network Accuracy from Weights
- Towards Task and Architecture-Independent Generalization Gap Predictors
- MetaFun: Meta-Learning with Iterative Functional Updates