Dictionary Learning over Distributed Models
arXiv:1402.1515 · doi:10.1109/TSP.2014.2385045
Abstract
In this paper, we consider learning dictionary models over a network of agents, where each agent is only in charge of a portion of the dictionary elements. This formulation is relevant in Big Data scenarios where large dictionary models may be spread over different spatial locations and it is not feasible to aggregate all dictionaries in one location due to communication and privacy considerations. We first show that the dual function of the inference problem is an aggregation of individual cost functions associated with different agents, which can then be minimized efficiently by means of diffusion strategies. The collaborative inference step generates dual variables that are used by the agents to update their dictionaries without the need to share these dictionaries or even the coefficient models for the training data. This is a powerful property that leads to an effective distributed procedure for learning dictionaries over large networks (e.g., hundreds of agents in our experiments). Furthermore, the proposed learning strategy operates in an online manner and is able to respond to streaming data, where each data sample is presented to the network once.
arXiv comment16 pages, 8 figures. To appear in IEEE Transactions on Signal Processing
References in corpus (13)
- Supervised Dictionary Learning
- Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
- Diffusion Strategies Outperform Consensus Strategies for Distributed Estimation over Adaptive Networks
- Distributed Constrained Optimization by Consensus-Based Primal-Dual Perturbation Method
- D-ADMM: A Communication-Efficient Distributed Algorithm For Separable Optimization
- Distributed Basis Pursuit
- Distributed Random Projection Algorithm for Convex Optimization
- Distributed Pareto Optimization via Diffusion Strategies
- PETRELS: Parallel Subspace Estimation and Tracking by Recursive Least Squares from Partial Observations
- Adaptive Penalty-Based Distributed Stochastic Convex Optimization
- On the Convergence of Decentralized Gradient Descent
- Discovering Emerging Topics in Social Streams via Link Anomaly Detection
- Distributed dictionary learning over a sensor network
Cited by in corpus (19)
- Cloud K-SVD: A Collaborative Dictionary Learning Algorithm for Big, Distributed Data
- Diffusion LMS for Multitask Problems with Local Linear Equality Constraints
- Supervised Learning Under Distributed Features
- Online Nonnegative Matrix Factorization with Outliers
- Exact Diffusion for Distributed Optimization and Learning --- Part I: Algorithm Development
- Improving the Sample and Communication Complexity for Decentralized Non-Convex Optimization: A Joint Gradient Estimation and Tracking Approach
- On the Learning Behavior of Adaptive Networks - Part I: Transient Analysis
- Walkman: A Communication-Efficient Random-Walk Algorithm for Decentralized Optimization
- Exact Diffusion for Distributed Optimization and Learning --- Part II: Convergence Analysis
- Online Nonnegative Matrix Factorization with General Divergences
- Asynchronous adaptive networks
- Differentially Private Decentralized Optimization with Relay Communication
- Decentralized Dictionary Learning Over Time-Varying Digraphs
- GT-STORM: Taming Sample, Communication, and Memory Complexities in Decentralized Non-Convex Learning
- Dynamic Average Diffusion with randomized Coordinate Updates
- On the Asymptotic Bias of the Diffusion-Based Distributed Pareto Optimization
- Distributed Low-Rank Adaptive Algorithms Based on Alternating Optimization and Applications
- Study of Diffusion Normalized Least Mean M-estimate Algorithms
- Strong Duality in Nonconvex Quadratic Problems with Separable Quadratic Constraints