Scaling the Wild: Decentralizing Hogwild!-style Shared-memory SGD

Bapi Chatterjee, Vyacheslav Kungurtsev, Dan Alistarh

Powered by the simplicity of lock-free asynchrony, Hogwilld! is a go-to approach to parallelize SGD over a shared-memory setting. Despite its popularity and concomitant extensions,…