Q-Sparse: All Large Language Models can be Fully Sparsely-Activated

Hongyu Wang, Shuming Ma, Ruiping Wang +1

We introduce, Q-Sparse, a simple yet effective approach to training sparsely-activated large language models (LLMs). Q-Sparse enables full sparsity of activations in LLMs which can…