Finding trainable sparse networks through Neural Tangent Transfer

Abstract

Deep neural networks have dramatically transformed machine learning, but their memory and energy demands are substantial. The requirements of real biological neural networks are rather modest in comparison, and one feature that might underlie this austerity is their sparse connectivity. In deep learning, trainable sparse networks that perform well on a specific task are usually constructed using label-dependent pruning criteria. In this article, we introduce Neural Tangent Transfer, a method that instead finds trainable sparse networks in a label-free manner. Specifically, we find sparse networks whose training dynamics, as characterized by the neural tangent kernel, mimic those of dense networks in function space. Finally, we evaluate our label-agnostic approach on several standard classification tasks and show that the resulting sparse networks achieve higher classification performance while converging faster.

Accepted by ICML 2020

References in corpus (8)

• Deep Learning in Neural Networks: An Overview
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Compressing Deep Convolutional Networks using Vector Quantization
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Memory Bounded Deep Convolutional Networks
• Enhanced Convolutional Neural Tangent Kernels
• Uncoupled Regression from Pairwise Comparison Data