Understanding Deflation Process in Over-parametrized Tensor Decomposition

Abstract

In this paper we study the training dynamics for gradient flow on over-parametrized tensor decomposition problems. Empirically, such training process often first fits larger components and then discovers smaller components, which is similar to a tensor deflation process that is commonly used in tensor decomposition algorithms. We prove that for orthogonally decomposable tensor, a slightly modified version of gradient flow would follow a tensor deflation process and recover all the tensor components. Our proof suggests that for orthogonal tensors, gradient flow dynamics works similarly as greedy low-rank learning in the matrix setting, which is a first step towards understanding the implicit regularization effect of over-parametrized models for low-rank tensors.

arXiv commentNeurIPS 2021 Camera Ready

References in corpus (23)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On Lazy Training in Differentiable Programming
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Learning One-hidden-layer Neural Networks with Landscape Design
• When Do Neural Networks Outperform Kernel Methods?
• Implicit Regularization in Deep Matrix Factorization
• Risk and parameter convergence of logistic regression
• Asymptotics of Wide Networks from Feynman Diagrams
• Implicit Bias of Gradient Descent on Linear Convolutional Networks
• Stochastic Particle Gradient Descent for Infinite Ensembles
• Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
• Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
• Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
• Towards Understanding Hierarchical Learning: Benefits of Neural Representations
• Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
• Implicit Regularization in Tensor Factorization