The Outer Product Structure of Neural Network Derivatives
arXiv:1810.03798
Abstract
In this paper, we show that feedforward and recurrent neural networks exhibit an outer product derivative structure but that convolutional neural networks do not. This structure makes it possible to use higher-order information without needing approximations or infeasibly large amounts of memory, and it may also provide insights into the geometry of neural network optima. The ability to easily access these derivatives also suggests a new, geometric approach to regularization. We then discuss how this structure could be used to improve training methods, increase network robustness and generalizability, and inform network compression methods.
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
- Towards Deep Neural Network Architectures Robust to Adversarial Examples
- Bayesian Compression for Deep Learning
- Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
- RES: Regularized Stochastic BFGS Algorithm
- A Linearly-Convergent Stochastic L-BFGS Algorithm
- Stronger generalization bounds for deep nets via a compression approach
- Distributed Machine Learning via Sufficient Factor Broadcasting
- Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form