Harnessing the Power of Serverless Runtimes for Large-Scale Optimization
arXiv:1901.03161
Abstract
The event-driven and elastic nature of serverless runtimes makes them a very efficient and cost-effective alternative for scaling up computations. So far, they have mostly been used for stateless, data parallel and ephemeral computations. In this work, we propose using serverless runtimes to solve generic, large-scale optimization problems. Specifically, we build a master-worker setup using AWS Lambda as the source of our workers, implement a parallel optimization algorithm to solve a regularized logistic regression problem, and show that relative speedups up to 256 workers and efficiencies above 70% up to 64 workers can be expected. We also identify possible algorithmic and system-level bottlenecks, propose improvements, and discuss the limitations and challenges in realizing these improvements.
arXiv comment9 pages, double column, 9 figures
References in corpus (6)
- Revisiting Distributed Synchronous SGD
- Asynchronous Distributed ADMM for Large-Scale Optimization- Part I: Algorithm and Convergence Analysis
- Asynchronous Distributed ADMM for Large-Scale Optimization- Part II: Linear Convergence Analysis and Numerical Performance
- numpywren: serverless linear algebra
- Hemingway: Modeling Distributed Optimization Algorithms
- Characterizing Task Completion Latencies in Fog Computing