DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

Tao Zhang, Jianchao Tan, Pingwei Sun +5

Softmax attention stores key and value vectors for every preceding token, causing inference memory to grow with sequence length. Recent language models incorporating Gated DeltaNet…