Fast Jacobian-Vector Product for Deep Networks

Abstract

Jacobian-vector products (JVPs) form the backbone of many recent developments in Deep Networks (DNs), with applications including faster constrained optimization, regularization with generalization guarantees, and adversarial example sensitivity assessments. Unfortunately, JVPs are computationally expensive for real world DN architectures and require the use of automatic differentiation to avoid manually adapting the JVP program when changing the DN architecture. We propose a novel method to quickly compute JVPs for any DN that employ Continuous Piecewise Affine (e.g., leaky-ReLU, max-pooling, maxout, etc.) nonlinearities. We show that our technique is on average $2\times$ faster than the fastest alternative over $13$ DN architectures and across various hardware. In addition, our solution does not require automatic differentiation and is thus easy to deploy in software, requiring only the modification of a few lines of codes that do not depend on the DN architecture.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Wide Residual Networks
• On the Number of Linear Regions of Deep Neural Networks
• Group Equivariant Convolutional Networks
• Deep Learning: A Critical Appraisal
• Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
• Understanding Deep Neural Networks with Rectified Linear Units
• On the number of response regions of deep feed forward networks with piece-wise linear activations
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Data augmentation instead of explicit regularization
• Imposing Hard Constraints on Deep Networks: Promises and Limitations
• Learning disconnected manifolds: a no GANs land
• Dissecting Deep Neural Networks
• Implicit Rugosity Regularization via Data Augmentation
• Max-Affine Spline Insights into Deep Generative Networks
• Hyperplane Arrangements of Trained ConvNets Are Biased
• The Geometry of Deep Networks: Power Diagram Subdivision