Varuna: Scalable, Low-cost Training of Massive Deep Learning Models

Sanjith Athlur, Nitika Saran, Muthian Sivathanu +2

Systems for training massive deep learning models (billions of parameters) today assume and require specialized "hyper-clusters": hundreds or thousands of GPUs wired with specializ…