Data-Dependent Path Normalization in Neural Networks
arXiv:1511.06747
Abstract
We propose a unified framework for neural net normalization, regularization and optimization, which includes Path-SGD and Batch-Normalization and interpolates between them across two different dimensions. Through this framework we investigate issue of invariance of the optimization, data dependence and the connection with natural gradients.
arXiv comment17 pages, 3 figures
References in corpus (6)
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- No More Pesky Learning Rates
- Path-SGD: Path-Normalized Optimization in Deep Neural Networks
- In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
- Norm-Based Capacity Control in Neural Networks
- Natural Neural Networks