Distributed Learning over Networks with Graph-Attention-Based Personalization

Abstract

In conventional distributed learning over a network, multiple agents collaboratively build a common machine learning model. However, due to the underlying non-i.i.d. data distribution among agents, the unified learning model becomes inefficient for each agent to process its locally accessible data. To address this problem, we propose a graph-attention-based personalized training algorithm (GATTA) for distributed deep learning. The GATTA enables each agent to train its local personalized model while exploiting its correlation with neighboring nodes and utilizing their useful information for aggregation. In particular, the personalized model in each agent is composed of a global part and a node-specific part. By treating each agent as one node in a graph and the node-specific parameters as its features, the benefits of the graph attention mechanism can be inherited. Namely, instead of aggregation based on averaging, it learns the specific weights for different neighboring nodes without requiring prior knowledge about the graph structure or the neighboring nodes' data distribution. Furthermore, relying on the weight-learning procedure, we develop a communication-efficient GATTA by skipping the transmission of information with small aggregation weights. Additionally, we theoretically analyze the convergence properties of GATTA for non-convex loss functions. Numerical results validate the excellent performances of the proposed algorithms in terms of convergence and communication cost.

arXiv commentAccepted for publication in IEEE TSP; with supplementary details for the derivations

References in corpus (15)

• A Comprehensive Survey on Graph Neural Networks
• Federated Learning: Challenges, Methods, and Future Directions
• Federated Learning with Non-IID Data
• Graph Convolutional Networks for Hyperspectral Image Classification
• An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• Learnable Manifold Alignment (LeMA) : A Semi-supervised Cross-modality Learning Framework for Land Cover and Land Use Classification
• Federated Learning of a Mixture of Global and Local Models
• Federated Evaluation of On-device Personalization
• An improved convergence analysis for decentralized online stochastic non-convex optimization
• Communication-Efficient Local Decentralized SGD Methods
• Distributed ADMM with Synergetic Communication and Computation
• Personalized Federated Learning with Exact Stochastic Gradient Descent