Gradients as Features for Deep Representation Learning

Abstract

We address the challenging problem of deep representation learning--the efficient adaption of a pre-trained deep network to different tasks. Specifically, we propose to explore gradient-based features. These features are gradients of the model parameters with respect to a task-specific loss given an input sample. Our key innovation is the design of a linear model that incorporates both gradient and activation of the pre-trained network. We show that our model provides a local linear approximation to an underlying deep model, and discuss important theoretical insights. Moreover, we present an efficient algorithm for the training and inference of our model without computing the actual gradient. Our method is evaluated across a number of representation-learning tasks on several datasets and using different network architectures. Strong results are obtained in all settings, and are well-aligned with our theoretical insights.

ICLR 2020 conference paper

References in corpus (11)

• Auto-Encoding Variational Bayes
• How transferable are features in deep neural networks?
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Density estimation using Real NVP
• Adversarial Feature Learning
• A Convergence Theory for Deep Learning via Over-Parameterization
• Adversarially Learned Inference
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Data-dependent Initializations of Convolutional Neural Networks
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Knowledge Transfer with Jacobian Matching

Cited by in corpus (6)

• Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
• Deep Active Learning by Leveraging Training Dynamics
• Predicting Training Time Without Training
• How Fine-Tuning Allows for Effective Meta-Learning
• Fast Adaptation with Linearized Neural Networks
• DIVA: Dataset Derivative of a Learning Task