First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models

Chi Ma, Mincong Huang, Ying Zhang +5

Dynamic activation (DA) techniques, such as DejaVu and MoEfication, have demonstrated their potential to significantly enhance the inference efficiency of large language models (LL…