1.3k citations · 12.3k across the 104 of their papers we have counts for
17 papers · 1 filter
Improving Generalization Performance by Switching from Adam to SGD
Nitish Shirish Keskar, Richard Socher
Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD…
A Flexible Approach to Automated RNN Architecture Generation
Martin Schrimpf, Stephen Merity, James Bradbury +1
The process of designing neural architectures requires expert knowledge and extensive trial and error. While automated architecture search may simplify these requirements, the recu…
Block-diagonal Hessian-free Optimization for Training Neural Networks
Huishuai Zhang, Caiming Xiong, James Bradbury +1
Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes…
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning
Tianmin Shu, Caiming Xiong, Richard Socher
Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-…
Improved Regularization Techniques for End-to-End Speech Recognition
Yingbo Zhou, Caiming Xiong, Richard Socher
Regularization is important for end-to-end speech models, since the models are highly flexible and easy to overfit. Data augmentation and dropout has been important for improving e…
Improving End-to-End Speech Recognition with Policy Learning
Yingbo Zhou, Caiming Xiong, Richard Socher
Connectionist temporal classification (CTC) is widely used for maximum likelihood learning in end-to-end speech recognition models. However, there is usually a disparity between th…