Dynamic Parameter Allocation in Parameter Servers

Abstract

To keep up with increasing dataset sizes and model complexity, distributed training has become a necessity for large machine learning tasks. Parameter servers ease the implementation of distributed parameter management---a key concern in distributed training---, but can induce severe communication overhead. To reduce communication overhead, distributed machine learning algorithms use techniques to increase parameter access locality (PAL), achieving up to linear speed-ups. We found that existing parameter servers provide only limited support for PAL techniques, however, and therefore prevent efficient training. In this paper, we explore whether and to what extent PAL techniques can be supported, and whether such support is beneficial. We propose to integrate dynamic parameter allocation into parameter servers, describe an efficient implementation of such a parameter server called Lapse, and experimentally compare its performance to existing parameter servers across a number of machine learning tasks. We found that Lapse provides near-linear scaling and can be orders of magnitude faster than existing parameter servers.

References in corpus (12)

• Efficient Estimation of Word Representations in Vector Space
• TensorFlow: A system for large-scale machine learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• A Review of Relational Machine Learning for Knowledge Graphs
• Complex Embeddings for Simple Link Prediction
• Analogical Inference for Multi-Relational Embeddings
• PyTorch-BigGraph: A Large-scale Graph Embedding System
• Priority-based Parameter Propagation for Distributed DNN Training
• NOMAD: Non-locking, stOchastic Multi-machine algorithm for Asynchronous and Decentralized matrix completion
• Dynamic Parameter Allocation in Parameter Servers
• Computing Web-scale Topic Models using an Asynchronous Parameter Server

Cited by in corpus (3)

• Towards Demystifying Serverless Machine Learning Training
• NuPS: A Parameter Server for Machine Learning with Non-Uniform Parameter Access
• Dynamic Parameter Allocation in Parameter Servers