Tensor Contraction Layers for Parsimonious Deep Nets

Abstract

Tensors offer a natural representation for many kinds of data frequently encountered in machine learning. Images, for example, are naturally represented as third order tensors, where the modes correspond to height, width, and channels. Tensor methods are noted for their ability to discover multi-dimensional dependencies, and tensor decompositions in particular, have been used to produce compact low-rank approximations of data. In this paper, we explore the use of tensor contractions as neural network layers and investigate several ways to apply them to activation tensors. Specifically, we propose the Tensor Contraction Layer (TCL), the first attempt to incorporate tensor contractions as end-to-end trainable neural network layers. Applied to existing networks, TCLs reduce the dimensionality of the activation tensors and thus the number of model parameters. We evaluate the TCL on the task of image recognition, augmenting two popular networks (AlexNet, VGG). The resulting models are trainable end-to-end. Applying the TCL to the task of image recognition, using the CIFAR100 and ImageNet datasets, we evaluate the effect of parameter reduction via tensor contraction on performance. We demonstrate significant model compression without significant impact on the accuracy and, in some cases, improved performance.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Tensorizing Neural Networks
• On the Expressive Power of Deep Learning: A Tensor Analysis
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Deep Multi-task Representation Learning: A Tensor Factorisation Approach
• Global Optimality in Tensor Factorization, Deep Learning, and Beyond
• Training Input-Output Recurrent Neural Networks through Spectral Methods

Cited by in corpus (6)

• Tensor Methods in Computer Vision and Deep Learning
• Improving Efficiency in Convolutional Neural Network with Multilinear Filters
• Spectral Learning on Matrices and Tensors
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• BT-Nets: Simplifying Deep Neural Networks via Block Term Decomposition
• Matrix and tensor decompositions for training binary neural networks