Regularization Techniques for Learning with Matrices

Abstract

There is growing body of learning problems for which it is natural to organize the parameters into matrix, so as to appropriately regularize the parameters under some matrix norm (in order to impose some more sophisticated prior knowledge). This work describes and analyzes a systematic method for constructing such matrix-based, regularization methods. In particular, we focus on how the underlying statistical properties of a given problem can help us decide which regularization function is appropriate. Our methodology is based on the known duality fact: that a function is strongly convex with respect to some norm if and only if its conjugate function is strongly smooth with respect to the dual norm. This result has already been found to be a key component in deriving and analyzing several learning algorithms. We demonstrate the potential of this framework by deriving novel generalization and regret bounds for multi-task learning, multi-class learning, and kernel learning.

References in corpus (2)

• Consistency of the group Lasso and multiple kernel learning
• Taking Advantage of Sparsity in Multi-Task Learning

Cited by in corpus (28)

• A Survey on Multi-Task Learning
• Large-scale Multi-label Learning with Missing Labels
• OptShrink: An algorithm for improved low-rank signal matrix denoising by optimal, data-driven singular value shrinkage
• The Benefit of Multitask Representation Learning
• Faster Rates for the Frank-Wolfe Method over Strongly-Convex Sets
• The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning
• Structured Sparsity and Generalization
• Towards minimax policies for online linear optimization with bandit feedback
• Fast Rates by Transferring from Auxiliary Hypotheses
• Distributed stochastic optimization via matrix exponential learning
• On the Generalization Ability of Online Learning Algorithms for Pairwise Loss Functions
• Sparse Logistic Regression Learns All Discrete Pairwise Graphical Models
• Online mirror descent and dual averaging: keeping pace in the dynamic case
• An Inequality with Applications to Structured Sparsity and Multitask Dictionary Learning
• A Generalized Online Mirror Descent with Applications to Classification and Regression
• Fast Optimization with Zeroth-Order Feedback in Distributed, Multi-User MIMO Systems
• Conditional gradient type methods for composite nonlinear and stochastic optimization
• On the convergence of mirror descent beyond stochastic convex programming
• A Provably Communication-Efficient Asynchronous Distributed Inference Method for Convex and Nonconvex Problems
• Bounds for Vector-Valued Function Estimation
• Generalized Stochastic Frank-Wolfe Algorithm with Stochastic "Substitute" Gradient for Structured Convex Optimization
• Online First-Order Framework for Robust Convex Optimization
• Gains and Losses are Fundamentally Different in Regret Minimization: The Sparse Case
• Implicit Regularization in Matrix Sensing via Mirror Descent
• Guaranteed Classification via Regularized Similarity Learning
• Multitask Online Mirror Descent
• Refined approachability algorithms and application to regret minimization with global costs
• Convergence of Online Mirror Descent