Tensor Regression Networks

Abstract

Convolutional neural networks typically consist of many convolutional layers followed by one or more fully connected layers. While convolutional layers map between high-order activation tensors, the fully connected layers operate on flattened activation vectors. Despite empirical success, this approach has notable drawbacks. Flattening followed by fully connected layers discards multilinear structure in the activations and requires many parameters. We address these problems by incorporating tensor algebraic operations that preserve multilinear structure at every layer. First, we introduce Tensor Contraction Layers (TCLs) that reduce the dimensionality of their input while preserving their multilinear structure using tensor contraction. Next, we introduce Tensor Regression Layers (TRLs), which express outputs through a low-rank multilinear mapping from a high-order activation tensor to an output tensor of arbitrary order. We learn the contraction and regression factors end-to-end, and produce accurate nets with fewer parameters. Additionally, our layers regularize networks by imposing low-rank constraints on the activations (TCL) and regression weights (TRL). Experiments on ImageNet show that, applied to VGG and ResNet architectures, TCLs and TRLs reduce the number of parameters compared to fully connected layers by more than 65% while maintaining or increasing accuracy. In addition to the space savings, our approach's ability to leverage topological structure can be crucial for structured data such as MRI. In particular, we demonstrate significant performance improvements over comparable architectures on three tasks associated with the UK Biobank dataset.

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Tensor Decomposition for Signal Processing and Machine Learning
• Tensor Decompositions for Signal Processing Applications From Two-way to Multiway Component Analysis
• Tensorizing Neural Networks
• Tensor Regression with Applications in Neuroimaging Data Analysis
• Convolutional neural networks with low-rank regularization
• Experimental Support for a Categorical Compositional Distributional Model of Meaning
• TensorLy: Tensor Learning in Python
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Deep Multi-task Representation Learning: A Tensor Factorisation Approach
• Global Optimality in Tensor Factorization, Deep Learning, and Beyond
• Learning from Multiway Data: Simple and Efficient Tensor Regression
• On the Expressive Power of Overlapping Architectures of Deep Learning
• Training Input-Output Recurrent Neural Networks through Spectral Methods

Cited by in corpus (9)

• TensorLy: Tensor Learning in Python
• Tensor Methods in Computer Vision and Deep Learning
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Spectral Learning on Matrices and Tensors
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Bayesian Tensorized Neural Networks with Automatic Rank Selection
• Linguistically-aware Attention for Reducing the Semantic-Gap in Vision-Language Tasks
• Matrix and tensor decompositions for training binary neural networks
• Higher-order Count Sketch: Dimensionality Reduction That Retains Efficient Tensor Operations