8 papers
Recurrence of Optimum for Training Weight and Activation Quantized Networks
Ziang Long, Penghang Yin, Jack Xin
Deep neural networks (DNNs) are quantized for efficient inference on resource-constrained platforms. However, training deep learning models with low-precision weights and activatio…
Learning Quantized Neural Nets by Coarse Gradient Method for Non-linear Classification
Ziang Long, Penghang Yin, Jack Xin
Quantized or low-bit neural networks are attractive due to their inference efficiency. However, training deep neural networks with quantized activations involves minimizing a disco…
Global Convergence and Geometric Characterization of Slow to Fast Weight Evolution in Neural Network Training for Classifying Linearly Non-Separable Data
Ziang Long, Penghang Yin, Jack Xin
In this paper, we study the dynamics of gradient descent in learning neural networks for classification problems. Unlike in existing works, we consider the linearly non-separable c…
Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
Penghang Yin, Jiancheng Lyu, Shuai Zhang +3
Training activation quantized neural networks involves minimizing a piecewise constant function whose gradient vanishes almost everywhere, which is undesirable for the standard bac…
Non-ergodic Convergence Analysis of Heavy-Ball Algorithms
Tao Sun, Penghang Yin, Dongsheng Li +3
In this paper, we revisit the convergence of the Heavy-ball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/…
Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization
Bao Wang, Alex T. Lin, Wei Zhu +3
We improve the robustness of Deep Neural Net (DNN) to adversarial attacks by using an interpolating function as the output activation. This data-dependent activation remarkably imp…