Memory and Bandwidth are All You Need for Fully Sharded Data Parallel

Jiangtao Wang, Jan Ebert, Oleg Filatov +1

Transformer models have revolutionized a wide spectrum of disciplines, especially in language processing. The recent success has proven that model size scalability is crucial for a…