Proximal Multitask Learning over Networks with Sparsity-inducing Coregularization
arXiv:1509.01360 · doi:10.1109/TSP.2016.2601282
Abstract
In this work, we consider multitask learning problems where clusters of nodes are interested in estimating their own parameter vector. Cooperation among clusters is beneficial when the optimal models of adjacent clusters have a good number of similar entries. We propose a fully distributed algorithm for solving this problem. The approach relies on minimizing a global mean-square error criterion regularized by non-differentiable terms to promote cooperation among neighboring clusters. A general diffusion forward-backward splitting strategy is introduced. Then, it is specialized to the case of sparsity promoting regularizers. A closed-form expression for the proximal operator of a weighted sum of -norms is derived to achieve higher efficiency. We also provide conditions on the step-sizes that ensure convergence of the algorithm in the mean and mean-square error sense. Simulations are conducted to illustrate the effectiveness of the strategy.
References in corpus (15)
- Gossip Algorithms for Distributed Signal Processing
- Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
- Diffusion Strategies Outperform Consensus Strategies for Distributed Estimation over Adaptive Networks
- l_0 Norm Constraint LMS Algorithm for Sparse System Identification
- Multitask Diffusion Adaptation over Networks
- Diffusion LMS over Multitask Networks
- Convergence Rate Analysis of Distributed Gossip (Linear Parameter) Estimation: Fundamental Limits and Tradeoffs
- Sparse Distributed Learning Based on Diffusion Adaptation
- Distributed Random Projection Algorithm for Convex Optimization
- Distributed Pareto Optimization via Diffusion Strategies
- Online Sparse System Identification and Signal Reconstruction using Projections onto Weighted Balls
- Distributed Clustering and Learning Over Networks
- Distributed Diffusion-Based LMS for Node-Specific Adaptive Parameter Estimation
- Estimation of Space-Time Varying Parameters Using a Diffusion LMS Algorithm
- Diffusion Adaptation Strategies for Distributed Estimation over Gaussian Markov Random Fields
Cited by in corpus (13)
- Multitask learning over graphs: An Approach for Distributed, Streaming Machine Learning
- Diffusion LMS for Multitask Problems with Local Linear Equality Constraints
- Multitask diffusion adaptation over networks with common latent representations
- On reducing the communication cost of the diffusion LMS algorithm
- Resilient Distributed Diffusion in Networks with Adversaries
- Dif-MAML: Decentralized Multi-Agent Meta-Learning
- Decentralized Sparse Multitask RLS over Networks
- Decentralized Multi-Task Learning Based on Extreme Learning Machines
- Asynchronous adaptive networks
- Affine Combination of Diffusion Strategies over Networks
- Online Distributed Learning over Graphs with Multitask Graph-Filter Models
- Privacy-Preserving Distributed Projection LMS for Linear Multitask Networks
- Study of Diffusion Normalized Least Mean M-estimate Algorithms