Dropout as a Low-Rank Regularizer for Matrix Factorization

Abstract

Regularization for matrix factorization (MF) and approximation problems has been carried out in many different ways. Due to its popularity in deep learning, dropout has been applied also for this class of problems. Despite its solid empirical performance, the theoretical properties of dropout as a regularizer remain quite elusive for this class of problems. In this paper, we present a theoretical analysis of dropout for MF, where Bernoulli random variables are used to drop columns of the factors. We demonstrate the equivalence between dropout and a fully deterministic model for MF in which the factors are regularized by the sum of the product of squared Euclidean norms of the columns. Additionally, we inspect the case of a variable sized factorization and we prove that dropout achieves the global minimum of a convex approximation problem with (squared) nuclear norm regularization. As a result, we conclude that dropout can be used as a low-rank regularizer with data dependent singular-value thresholding.

References in corpus (3)

• Adam: A Method for Stochastic Optimization
• Improving neural networks by preventing co-adaptation of feature detectors
• Adding noise to the input of a model trained with a regularized objective

Cited by in corpus (10)

• Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks
• Deep matrix factorizations
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• The Implicit and Explicit Regularization Effects of Dropout
• Dropout: Explicit Forms and Capacity Control
• Almost Sure Convergence of Dropout Algorithms for Neural Networks
• An ETF view of Dropout regularization
• Dropout as a Regularizer of Interaction Effects
• Regularization in ResNet with Stochastic Depth
• Comments on Efficient Singular Value Thresholding Computation