238 citations · 694 across the 33 of their papers we have counts for
9 papers · 1 filter
PARQ: Piecewise-Affine Regularized Quantization
Lisa Jin, Jianhao Ma, Zechun Liu +3
We develop a principled method for quantization-aware training (QAT) of large-scale machine learning models. Specifically, we show that convex, piecewise-affine regularization (PAR…
Efficient and Robust Quantization-aware Training via Adaptive Coreset Selection
Xijie Huang, Zechun Liu, Shih-Yang Liu +1
Quantization-aware training (QAT) is a representative model compression method to reduce redundancy in weights and activations. However, most existing QAT methods require end-to-en…
SDQ: Stochastic Differentiable Quantization with Mixed Precision
Xijie Huang, Zhiqiang Shen, Shichao Li +5
In order to deploy deep models in a computationally efficient manner, model quantization approaches have been frequently used. In addition, as new hardware that supports mixed bitw…
BiT: Robustly Binarized Multi-distilled Transformer
Zechun Liu, Barlas Oguz, Aasish Pappu +5
Modern pre-trained transformers have rapidly advanced the state-of-the-art in machine learning, but have also grown in parameters and computational complexity, making them increasi…
Data-Free Neural Architecture Search via Recursive Label Calibration
Zechun Liu, Zhiqiang Shen, Yun Long +3
This paper aims to explore the feasibility of neural architecture search (NAS) given only a pre-trained model without using any original training data. This is an important circums…
How Do Adam and Training Strategies Help BNNs Optimization?
Zechun Liu, Zhiqiang Shen, Shichao Li +3
The best performing Binary Neural Networks (BNNs) are usually attained using Adam optimization and its multi-step training variants. However, to the best of our knowledge, few stud…
"BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
Tianlong Chen, Zhenyu Zhang, Xu Ouyang +3
Batch normalization (BN) is a key facilitator and considered essential for state-of-the-art binary neural networks (BNN). However, the BN layer is costly to calculate and is typica…
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
Zhiqiang Shen, Zechun Liu, Dejia Xu +3
This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation. We begin by introducing the motivation beh…
Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
Koen Helwegen, James Widdicombe, Lukas Geiger +3
Optimization of Binarized Neural Networks (BNNs) currently relies on real-valued latent weights to accumulate small update steps. In this paper, we argue that these latent weights…