Bayesian Tensorized Neural Networks with Automatic Rank Selection

Abstract

Tensor decomposition is an effective approach to compress over-parameterized neural networks and to enable their deployment on resource-constrained hardware platforms. However, directly applying tensor compression in the training process is a challenging task due to the difficulty of choosing a proper tensor rank. In order to achieve this goal, this paper proposes a Bayesian tensorized neural network. Our Bayesian method performs automatic model compression via an adaptive tensor rank determination. We also present approaches for posterior density calculation and maximum a posteriori (MAP) estimation for the end-to-end training of our tensorized neural network. We provide experimental validation on a fully connected neural network, a CNN and a residual neural network where our work produces $7.4\times$ to $137\times$ more compact neural networks directly from the training.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
• Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination
• Bayesian Compression for Deep Learning
• Alternating minimal energy methods for linear systems in higher dimensions. Part I: SPD systems
• Alternating minimal energy methods for linear systems in higher dimensions. Part II: Faster algorithm and application to nonsymmetric systems
• Bayesian Robust Tensor Factorization for Incomplete Multiway Data
• Ultimate tensorization: compressing convolutional and FC layers alike
• Rank regularization and Bayesian inference for tensor completion and extrapolation
• Tensor Regression Networks
• Soft Weight-Sharing for Neural Network Compression
• Long-term Forecasting using Higher Order Tensor RNNs
• Wide Compression: Tensor Ring Nets
• Compression and Interpretability of Deep Neural Networks via Tucker Tensor Layer: From First Principles to Tensor Valued Back-Propagation
• Robust Factorization and Completion of Streaming Tensor Data via Variational Bayesian Inference

Cited by in corpus (11)

• Block-Term Tensor Decomposition Model Selection and Computation: The Bayesian Way
• Tensor networks for interpretable and efficient quantum-inspired machine learning
• TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models
• Sparse Tucker Tensor Decomposition on a Hybrid FPGA-CPU Platform
• Active Subspace of Neural Networks: Structural Analysis and Universal Attacks
• 3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration
• Tensor Methods for Generating Compact Uncertainty Quantification and Deep Learning Models
• Low-Rank+Sparse Tensor Compression for Neural Networks
• Tensor Yard: One-Shot Algorithm of Hardware-Friendly Tensor-Train Decomposition for Convolutional Neural Networks
• MARS: Masked Automatic Ranks Selection in Tensor Decompositions
• Tucker Tensor Decomposition on FPGA