TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly

Toshiaki Koike-Akino, Jing Liu, Ye Wang

To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these methods hi…