The no-free-lunch theorems of supervised learning
arXiv:2202.04513 · doi:10.1007/s11229-021-03233-1
Abstract
The no-free-lunch theorems promote a skeptical conclusion that all possible machine learning algorithms equally lack justification. But how could this leave room for a learning theory, that shows that some algorithms are better than others? Drawing parallels to the philosophy of induction, we point out that the no-free-lunch results presuppose a conception of learning algorithms as purely data-driven. On this conception, every algorithm must have an inherent inductive bias, that wants justification. We argue that many standard learning algorithms should rather be understood as model-dependent: in each application they also require for input a model, representing a bias. Generic algorithms themselves, they can be given a model-relative justification.
References in corpus (10)
- Understanding deep learning requires rethinking generalization
- A Closer Look at Memorization in Deep Networks
- Exploring Generalization in Deep Learning
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Spectrally-normalized margin bounds for neural networks
- Norm-Based Capacity Control in Neural Networks
- Minimum Description Length Revisited
- Fast Rates for General Unbounded Loss Functions: from ERM to Generalized Bayes
- Absolutely No Free Lunches!
- Minimizers of the Empirical Risk and Risk Monotonicity