High-Performance Distributed ML at Scale through Parameter Server Consistency Models

Abstract

As Machine Learning (ML) applications increase in data size and model complexity, practitioners turn to distributed clusters to satisfy the increased computational and memory demands. Unfortunately, effective use of clusters for ML requires considerable expertise in writing distributed code, while highly-abstracted frameworks like Hadoop have not, in practice, approached the performance seen in specialized ML implementations. The recent Parameter Server (PS) paradigm is a middle ground between these extremes, allowing easy conversion of single-machine parallel ML applications into distributed ones, while maintaining high throughput through relaxed "consistency models" that allow inconsistent parameter reads. However, due to insufficient theoretical study, it is not clear which of these consistency models can really ensure correct ML algorithm output; at the same time, there remain many theoretically-motivated but undiscovered opportunities to maximize computational throughput. Motivated by this challenge, we study both the theoretical guarantees and empirical behavior of iterative-convergent ML algorithms in existing PS consistency models. We then use the gleaned insights to improve a consistency model using an "eager" PS communication mechanism, and implement it as a new PS system that enables ML algorithms to reach their solution more quickly.

19 pages, 2 figures

References in corpus (2)

• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Primitives for Dynamic Big Model Parallelism

Cited by in corpus (10)

• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
• Distributed Proximal Gradient Algorithm for Partially Asynchronous Computer Clusters
• Distributed Machine Learning via Sufficient Factor Broadcasting
• Strategies and Principles of Distributed Machine Learning on Big Data
• Petuum: A New Platform for Distributed Machine Learning on Big Data
• Diversity/Parallelism Trade-off in Distributed Systems with Redundancy
• Dynamic Stale Synchronous Parallel Distributed Training for Deep Learning
• AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
• On the Performance and Convergence of Distributed Stream Processing via Approximate Fault Tolerance