90 citations · 162 across the 13 of their papers we have counts for
13 papers
Fine-grained Policy-driven I/O Sharing for Burst Buffers
Ed Karrels, Lei Huang, Yuhong Kan +5
A burst buffer is a common method to bridge the performance gap between the I/O needs of modern supercomputing applications and the performance of the shared file system on large-s…
Modulate Your Spectrum in Self-Supervised Learning
Xi Weng, Yunhao Ni, Tengwei Song +5
Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning (SSL) with joint embedding architectures. Typically, it involves a hard whitening…
A Lightweight Recurrent Learning Network for Sustainable Compressed Sensing
Yu Zhou, Yu Chen, Xiao Zhang +3
Recently, deep learning-based compressed sensing (CS) has achieved great success in reducing the sampling and computational cost of sensing systems and improving the reconstruction…
FIXED: Frustratingly Easy Domain Generalization with Mixup
Wang Lu, Jindong Wang, Han Yu +4
Domain generalization (DG) aims to learn a generalizable model from multiple training domains such that it can perform well on unseen target domains. A popular strategy is to augme…
Understanding the Failure of Batch Normalization for Transformers in NLP
Jiaxi Wang, Ji Wu, Lei Huang
Batch Normalization (BN) is a core and prevalent technique in accelerating the training of deep neural networks and improving the generalization on Computer Vision (CV) tasks. Howe…
Bi-level Doubly Variational Learning for Energy-based Latent Variable Models
Ge Kan, Jinhu Lü, Tian Wang +5
Energy-based latent variable models (EBLVMs) are more expressive than conventional energy-based models. However, its potential on visual tasks are limited by its training process b…
Delving into the Estimation Shift of Batch Normalization in a Network
Lei Huang, Yi Zhou, Tian Wang +2
Batch normalization (BN) is a milestone technique in deep learning. It normalizes the activation using mini-batch statistics during training but the estimated population statistics…
Controllable Orthogonalization in Training DNNs
Lei Huang, Li Liu, Fan Zhu +4
Orthogonality is widely used for training deep neural networks (DNNs) due to its ability to maintain all singular values of the Jacobian close to 1 and reduce redundancy in represe…
An Investigation into the Stochasticity of Batch Whitening
Lei Huang, Lei Zhao, Yi Zhou +3
Batch Normalization (BN) is extensively employed in various network architectures by performing standardization within mini-batches. A full understanding of the process has been a…
Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
Lei Huang, Jie Qin, Li Liu +2
Conditioning analysis uncovers the landscape of an optimization objective by exploring the spectrum of its curvature matrix. This has been well explored theoretically for linear mo…
Iterative Normalization: Beyond Standardization towards Efficient Whitening
Lei Huang, Yi Zhou, Fan Zhu +2
Batch Normalization (BN) is ubiquitously employed for accelerating neural network training and improving the generalization capability by performing standardization within mini-bat…
Decorrelated Batch Normalization
Lei Huang, Dawei Yang, Bo Lang +1
Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batc…
Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
Lei Huang, Xianglong Liu, Bo Lang +3
Orthogonal matrix has shown advantages in training Recurrent Neural Networks (RNNs), but such matrix is limited to be square for the hidden-to-hidden transformation in RNNs. In thi…