175 citations · 192 across the 4 of their papers we have counts for
4 papers · 1 filter
MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference
Junfeng Wu, Zehao Fan, Hadjer Benmeziane +3
Mixture-of-experts (MoE) language models often exceed the memory of a single GPU. Expert offloading keeps most experts in host memory and loads them on demand, so decoding speed de…
Transformer Acceleration with Dynamic Sparse Attention
Liu Liu, Zheng Qu, Zhaodong Chen +2
Transformers are the mainstream of NLP applications and are becoming increasingly popular in other domains such as Computer Vision. Despite the improvements in model quality, the e…
Computation on Sparse Neural Networks: an Inspiration for Future Hardware
Fei Sun, Minghai Qin, Tianyun Zhang +3
Neural network models are widely used in solving many challenging problems, such as computer vision, personalized recommendation, and natural language processing. Those models are…
L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
Shuang Wu, Guoqi Li, Lei Deng +3
Batch Normalization (BN) has been proven to be quite effective at accelerating and improving the training of deep neural networks (DNNs). However, BN brings additional computation,…