Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps

Abstract

Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference speed and reduce memory usage compared to general linear maps. However, choosing which of the myriad structured transformations to use (and its associated parameterization) is a laborious task that requires trading off speed, space, and accuracy. We consider a different approach: we introduce a family of matrices called kaleidoscope matrices (K-matrices) that provably capture any structured matrix with near-optimal space (parameter) and time (arithmetic operation) complexity. We empirically validate that K-matrices can be automatically learned within end-to-end pipelines to replace hand-crafted procedures, in order to improve model quality. For example, replacing channel shuffles in ShuffleNet improves classification accuracy on ImageNet by up to 5%. K-matrices can also simplify hand-engineered pipelines -- we replace filter bank feature computation in speech data preprocessing with a learnable kaleidoscope layer, resulting in only 0.4% loss in accuracy on the TIMIT speech recognition task. In addition, K-matrices can capture latent structure in models: for a challenging permuted image classification task, a K-matrix based representation of permutations is able to learn the right latent structure and improves accuracy of a downstream convolutional model by over 9%. We provide a practically efficient implementation of our approach, and use K-matrices in a Transformer network to attain 36% faster end-to-end inference speed on a language translation task.

International Conference on Learning Representations (ICLR) 2020 spotlight

References in corpus (10)

• To prune, or not to prune: exploring the efficacy of pruning for model compression
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Full-Capacity Unitary Recurrent Neural Networks
• Orthogonal Random Features
• Compact Nonlinear Maps and Circulant Extensions
• Fast Approximation of Rotations and Hessians matrices
• AutoShuffleNet: Learning Permutation Matrices via an Exact Lipschitz Continuous Penalty in Deep Convolutional Neural Networks