57 citations · 301 across the 27 of their papers we have counts for
8 papers · 1 filter
Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization
Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand +6
The early phase of training a deep neural network has a dramatic effect on the local curvature of the loss function. For instance, using a small learning rate does not guarantee st…
Neural Pruning via Growing Regularization
Huan Wang, Can Qin, Yulun Zhang +1
Regularization has long been utilized to learn sparsity in deep neural network pruning. However, its role is mainly explored in the small penalty strength regime. In this work, we…
Unsupervised Paraphrasing with Pretrained Language Models
Tong Niu, Semih Yavuz, Yingbo Zhou +3
Paraphrase generation has benefited extensively from recent progress in the designing of training objectives and model architectures. However, previous explorations have largely fo…
Towards Understanding Hierarchical Learning: Benefits of Neural Representations
Minshuo Chen, Yu Bai, Jason D. Lee +4
Deep neural networks can empirically perform efficient hierarchical learning, in which the layers learn useful representations of the data. However, how they make use of the interm…
MNN: A Universal and Efficient Inference Engine
Xiaotang Jiang, Huan Wang, Yiliu Chen +9
Deploying deep learning models on mobile devices draws more and more attention recently. However, designing an efficient inference engine on devices is under the great challenges o…
Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
Yu Bai, Ben Krause, Huan Wang +2
We propose \emph{Taylorized training} as an initiative towards better understanding neural network training at finite width. Taylorized training involves training the -th order…
Neural Bayes: A Generic Parameterization Method for Unsupervised Representation Learning
Devansh Arpit, Huan Wang, Caiming Xiong +2
We introduce a parameterization method called Neural Bayes which allows computing statistical quantities that are in general difficult to compute and opens avenues for formulating…
Contradictory Structure Learning for Semi-supervised Domain Adaptation
Can Qin, Lichen Wang, Qianqian Ma +3
Current adversarial adaptation methods attempt to align the cross-domain features, whereas two challenges remain unsolved: 1) the conditional distribution mismatch and 2) the bias…