Short-circuiting Rings for Low-Latency AllReduce

Sarah-Michelle Hammer, Stefan Schmid, Rachee Singh +1

Efficient collective communication is critical for many distributed ML and HPC applications. In this context, it is widely believed that the Ring algorithm for the AllReduce collec…