Data-Dependent Path Normalization in Neural Networks

Abstract

We propose a unified framework for neural net normalization, regularization and optimization, which includes Path-SGD and Batch-Normalization and interpolates between them across two different dimensions. Through this framework we investigate issue of invariance of the optimization, data dependence and the connection with natural gradients.

17 pages, 3 figures

References in corpus (6)

• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• No More Pesky Learning Rates
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Norm-Based Capacity Control in Neural Networks
• Natural Neural Networks

Cited by in corpus (2)

• Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced
• Towards Better Generalization: BP-SVRG in Training Deep Neural Networks