270 citations · 599 across the 36 of their papers we have counts for
8 papers · 1 filter
Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precision
Xijie Huang, Zhiqiang Shen, Pingcheng Dong +1
Despite the outstanding performance of transformers in both language and vision tasks, the expanding computation and model size have increased the demand for efficient deployment.…
One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
Arnav Chavan, Zhuang Liu, Deepak Gupta +2
We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized pro…
Dropout Reduces Underfitting
Zhuang Liu, Zhiqiu Xu, Joseph Jin +2
Introduced by Hinton et al. in 2012, dropout has stood the test of time as a regularizer for preventing overfitting in neural networks. In this study, we demonstrate that dropout c…
SDQ: Stochastic Differentiable Quantization with Mixed Precision
Xijie Huang, Zhiqiang Shen, Shichao Li +5
In order to deploy deep models in a computationally efficient manner, model quantization approaches have been frequently used. In addition, as new hardware that supports mixed bitw…
Data-Free Neural Architecture Search via Recursive Label Calibration
Zechun Liu, Zhiqiang Shen, Yun Long +3
This paper aims to explore the feasibility of neural architecture search (NAS) given only a pre-trained model without using any original training data. This is an important circums…
How Do Adam and Training Strategies Help BNNs Optimization?
Zechun Liu, Zhiqiang Shen, Shichao Li +3
The best performing Binary Neural Networks (BNNs) are usually attained using Adam optimization and its multi-step training variants. However, to the best of our knowledge, few stud…