Revisiting LARS for Large Batch Training Generalization of Neural Networks

Khoi Do, Duong Nguyen, Hoa Nguyen +3

This paper explores Large Batch Training techniques using layer-wise adaptive scaling ratio (LARS) across diverse settings, uncovering insights. LARS algorithms with warm-up tend t…