Model Aggregation via Good-Enough Model Spaces
arXiv:1805.07782
Abstract
In many applications, the training data for a machine learning task is partitioned across multiple nodes, and aggregating this data may be infeasible due to communication, privacy, or storage constraints. Existing distributed optimization methods for learning global models in these settings typically aggregate local updates from each node in an iterative fashion. However, these approaches require many rounds of communication between nodes, and assume that updates can be synchronously shared across a connected network. In this work, we present Good-Enough Model Spaces (GEMS), a novel framework for learning a global model by carefully intersecting the sets of "good-enough" models across each node. Our approach utilizes minimal communication and does not require sharing of data between nodes. We present methods for learning both convex models and neural networks within this framework and discuss how small samples of held-out data can be used for post-learning fine-tuning. In experiments on image and medical datasets, our approach on average improves upon other baseline aggregation techniques such as ensembling or model averaging by as much as 15 points (accuracy).
arXiv comment21 pages, 6 figures, 8 tablees
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Overcoming catastrophic forgetting in neural networks
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Optimal Distributed Online Prediction using Mini-Batches
- Communication Efficient Distributed Optimization using an Approximate Newton-type Method
- Communication Complexity of Distributed Convex Learning and Optimization