Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees

Jue Wang, Binhang Yuan, Luka Rimanic +5

Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive…