Supervised Learning Under Distributed Features

Abstract

This work studies the problem of learning under both large datasets and large-dimensional feature space scenarios. The feature information is assumed to be spread across agents in a network, where each agent observes some of the features. Through local cooperation, the agents are supposed to interact with each other to solve an inference problem and converge towards the global minimizer of an empirical risk. We study this problem exclusively in the primal domain, and propose new and effective distributed solutions with guaranteed convergence to the minimizer with linear rate under strong convexity. This is achieved by combining a dynamic diffusion construction, a pipeline strategy, and variance-reduced techniques. Simulation results illustrate the conclusions.

References in corpus (19)

• A Survey on Multi-view Learning
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• On the Linear Convergence of the ADMM in Decentralized Consensus Optimization
• Multi-Agent Distributed Optimization via Inexact Consensus ADMM
• Distributed Constrained Optimization by Consensus-Based Primal-Dual Perturbation Method
• ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates
• A linear algorithm for optimization over directed graphs with geometric convergence
• Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization
• Convergence Rate Analysis of Distributed Gossip (Linear Parameter) Estimation: Fundamental Limits and Tradeoffs
• Distributed Basis Pursuit
• Distributed Pareto Optimization via Diffusion Strategies
• DSA: Decentralized Double Stochastic Averaging Gradient Algorithm
• Distributed Learning for Stochastic Generalized Nash Equilibrium Problems
• Dictionary Learning over Distributed Models
• SDCA without Duality, Regularization, and Individual Convexity
• Stochastic Learning under Random Reshuffling with Constant Step-sizes
• A Push-Pull Gradient Method for Distributed Optimization in Networks
• Coordinate-Descent Diffusion Learning by Networked Agents
• Exact Diffusion for Distributed Optimization and Learning --- Part I: Algorithm Development

Cited by in corpus (12)

• A Sharp Estimate on the Transient Time of Distributed Stochastic Gradient Descent
• Learning Privately over Distributed Features: An ADMM Sharing Approach
• A Unified Algorithmic Framework for Distributed Adaptive Signal and Feature Fusion Problems -- Part I: Algorithm Derivation
• Asymptotic Network Independence in Distributed Stochastic Optimization for Machine Learning
• Distributed Learning on Heterogeneous Resource-Constrained Devices
• The Internet of Things as a Deep Neural Network
• Decentralized Non-Convex Learning with Linearly Coupled Constraints
• Dynamic Average Diffusion with randomized Coordinate Updates
• Network Classifiers With Output Smoothing
• Secure Distributed On-Device Learning Networks With Byzantine Adversaries
• Distributed Dual Coordinate Ascent in General Tree Networks and Communication Network Effect on Synchronous Machine Learning
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications