Tensor Methods for Generating Compact Uncertainty Quantification and Deep Learning Models

Abstract

Tensor methods have become a promising tool to solve high-dimensional problems in the big data era. By exploiting possible low-rank tensor factorization, many high-dimensional model-based or data-driven problems can be solved to facilitate decision making or machine learning. In this paper, we summarize the recent applications of tensor computation in obtaining compact models for uncertainty quantification and deep learning. In uncertainty analysis where obtaining data samples is expensive, we show how tensor methods can significantly reduce the simulation or measurement cost. To enable the deployment of deep learning on resource-constrained hardware platforms, tensor methods can be used to significantly compress an over-parameterized neural network model or directly train a small-size model from scratch via optimization or statistical techniques. Recent Bayesian tensorized neural networks can automatically determine their tensor ranks in the training process.

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• Tensor Decomposition for Signal Processing and Machine Learning
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Tensor Decompositions for Signal Processing Applications From Two-way to Multiway Component Analysis
• Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination
• Tensorizing Neural Networks
• Stochastic Testing Method for Transistor-Level Uncertainty Quantification Based on Generalized Polynomial Chaos
• Compressing Recurrent Neural Network with Tensor Train
• Enabling High-Dimensional Hierarchical Uncertainty Quantification by ANOVA and Tensor-Train Decomposition
• Ultimate tensorization: compressing convolutional and FC layers alike
• Stochastic Testing Simulator for Integrated Circuits and MEMS: Hierarchical and Sparse Techniques
• Compression and Interpretability of Deep Neural Networks via Tucker Tensor Layer: From First Principles to Tensor Valued Back-Propagation
• Bayesian Tensorized Neural Networks with Automatic Rank Selection