26 citations · 93 across the 11 of their papers we have counts for
10 papers · 1 filter
On the Overlooked Structure of Stochastic Gradients
Zeke Xie, Qian-Yuan Tang, Mingming Sun +1
Stochastic gradients closely relate to both optimization and generalization of deep neural networks (DNNs). Some works attempted to explain the success of stochastic optimization f…
Sparse Double Descent: Where Network Pruning Aggravates Overfitting
Zheng He, Zeke Xie, Quanzhi Zhu +1
People usually believe that network pruning not only reduces the computational cost of deep networks, but also prevents overfitting by decreasing model capacity. However, our work…
Dataset Pruning: Reducing Training Data by Examining Generalization Influence
Shuo Yang, Zeke Xie, Hanyu Peng +3
The great success of deep learning heavily relies on increasingly larger training data, which comes at a price of huge computational and infrastructural costs. This poses crucial q…
On the Power-Law Hessian Spectrums in Deep Learning
Zeke Xie, Qian-Yuan Tang, Yunfeng Cai +2
It is well-known that the Hessian of deep loss landscape matters to optimization, generalization, and even robustness of deep learning. Recent works empirically discovered that the…
Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
Zeke Xie, Li Yuan, Zhanxing Zhu +1
It is well-known that stochastic gradient noise (SGN) acts as implicit regularization for deep learning and is essentially important for both optimization and generalization of dee…
On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective
Zeke Xie, Zhiqiang Xu, Jingzhao Zhang +2
Weight decay is a simple yet powerful regularization technique that has been very widely used in training of deep neural networks (DNNs). While weight decay has attracted much atte…
Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting
Zeke Xie, Fengxiang He, Shaopeng Fu +3
Deep learning is often criticized by two serious issues which rarely exist in natural nervous systems: overfitting and catastrophic forgetting. It can even memorize randomly labell…
Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum
Zeke Xie, Xinrui Wang, Huishuai Zhang +2
Adaptive Moment Estimation (Adam), which combines Adaptive Learning Rate and Momentum, would be the most popular stochastic optimizer for accelerating the training of deep neural n…
A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
Zeke Xie, Issei Sato, Masashi Sugiyama
Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice. SGD is known to find a flat minimum that often generalizes well. H…
A Quantum-Inspired Ensemble Method and Quantum-Inspired Forest Regressors
Zeke Xie, Issei Sato
We propose a Quantum-Inspired Subspace(QIS) Ensemble Method for generating feature ensembles based on feature selections. We assign each principal component a Fraction Transition P…