Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm

Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3

Training large language models (LLMs) is highly memory-intensive, as training must store not only weights and optimizer states but also intermediate activations for backpropagation…