Dictionary Learning over Distributed Models

Abstract

In this paper, we consider learning dictionary models over a network of agents, where each agent is only in charge of a portion of the dictionary elements. This formulation is relevant in Big Data scenarios where large dictionary models may be spread over different spatial locations and it is not feasible to aggregate all dictionaries in one location due to communication and privacy considerations. We first show that the dual function of the inference problem is an aggregation of individual cost functions associated with different agents, which can then be minimized efficiently by means of diffusion strategies. The collaborative inference step generates dual variables that are used by the agents to update their dictionaries without the need to share these dictionaries or even the coefficient models for the training data. This is a powerful property that leads to an effective distributed procedure for learning dictionaries over large networks (e.g., hundreds of agents in our experiments). Furthermore, the proposed learning strategy operates in an online manner and is able to respond to streaming data, where each data sample is presented to the network once.

16 pages, 8 figures. To appear in IEEE Transactions on Signal Processing

References in corpus (9)

• Supervised Dictionary Learning
• Diffusion Strategies Outperform Consensus Strategies for Distributed Estimation over Adaptive Networks
• Distributed Constrained Optimization by Consensus-Based Primal-Dual Perturbation Method
• Distributed Random Projection Algorithm for Convex Optimization
• Distributed Pareto Optimization via Diffusion Strategies
• PETRELS: Parallel Subspace Estimation and Tracking by Recursive Least Squares from Partial Observations
• Adaptive Penalty-Based Distributed Stochastic Convex Optimization
• On the Convergence of Decentralized Gradient Descent
• Distributed dictionary learning over a sensor network

Cited by in corpus (19)

• Cloud K-SVD: A Collaborative Dictionary Learning Algorithm for Big, Distributed Data
• Diffusion LMS for Multitask Problems with Local Linear Equality Constraints
• Supervised Learning Under Distributed Features
• Online Nonnegative Matrix Factorization with Outliers
• Exact Diffusion for Distributed Optimization and Learning --- Part I: Algorithm Development
• Improving the Sample and Communication Complexity for Decentralized Non-Convex Optimization: A Joint Gradient Estimation and Tracking Approach
• On the Learning Behavior of Adaptive Networks - Part I: Transient Analysis
• Walkman: A Communication-Efficient Random-Walk Algorithm for Decentralized Optimization
• Exact Diffusion for Distributed Optimization and Learning --- Part II: Convergence Analysis
• Online Nonnegative Matrix Factorization with General Divergences
• Asynchronous adaptive networks
• Differentially Private Decentralized Optimization with Relay Communication
• Decentralized Dictionary Learning Over Time-Varying Digraphs
• GT-STORM: Taming Sample, Communication, and Memory Complexities in Decentralized Non-Convex Learning
• Dynamic Average Diffusion with randomized Coordinate Updates
• On the Asymptotic Bias of the Diffusion-Based Distributed Pareto Optimization
• Distributed Low-Rank Adaptive Algorithms Based on Alternating Optimization and Applications
• Study of Diffusion Normalized Least Mean M-estimate Algorithms
• Strong Duality in Nonconvex Quadratic Problems with Separable Quadratic Constraints