Effective Regularization Through Loss-Function Metalearning
arXiv:2010.00788
Abstract
Evolutionary computation can be used to optimize several different aspects of neural network architectures. For instance, the TaylorGLO method discovers novel, customized loss functions, resulting in improved performance, faster training, and improved data utilization. A likely reason is that such functions discourage overfitting, leading to effective regularization. This paper demonstrates theoretically that this is indeed the case for TaylorGLO. Learning rule decomposition reveals that evolved loss functions balance two factors: the pull toward zero error, and a push away from it to avoid overfitting. This is a general principle that may be used to understand other regularization techniques as well (as demonstrated in this paper for label smoothing). The theoretical analysis leads to a constraint that can be utilized to find more effective loss functions in practice; the mechanism also results in networks that are more robust (as demonstrated in this paper with adversarial inputs). The analysis in this paper thus constitutes a first step towards understanding regularization, and demonstrates the power of evolutionary neural architecture search in general.
A shorter version of this paper appeared in CEC 2025; this paper includes appendices, expanded references, and corrections
References in corpus (12)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Improved Regularization of Convolutional Neural Networks with Cutout
- Wide Residual Networks
- Reconciling modern machine learning practice and the bias-variance trade-off
- Path-SGD: Path-Normalized Optimization in Deep Neural Networks
- Evolved Policy Gradients
- Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
- Meta-Learning Update Rules for Unsupervised Representation Learning
- Geometry of Optimization and Implicit Regularization in Deep Learning
- Do We Need Zero Training Loss After Achieving Zero Training Error?
- Online Meta-Critic Learning for Off-Policy Actor-Critic Methods
- Learning State-Dependent Losses for Inverse Dynamics Learning