3 papers
cs.AI2024
Understanding Artificial Neural Network's Behavior from Neuron Activation Perspective
Yizhou Zhang, Yang Sui
This paper explores the intricate behavior of deep neural networks (DNNs) through the lens of neuron activation dynamics. We propose a probabilistic framework that can analyze mode…
cs.LG2024
MoE-I: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
Cheng Yang, Yang Sui, Jinqi Xiao +7
The emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. Compared to traditional LLMs, MoE LLMs outperform traditional LLMs by…
cs.CV2024
ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks
Yang Sui, Miao Yin, Yu Gong +3
Low-rank compression, a popular model compression technique that produces compact convolutional neural networks (CNNs) with low rankness, has been well-studied in the literature. O…