789 citations · 3.9k across the 31 of their papers we have counts for
4 papers · 1 filter
Improving Generalization Performance by Switching from Adam to SGD
Nitish Shirish Keskar, Richard Socher
Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD…
Weighted Transformer Network for Machine Translation
Karim Ahmed, Nitish Shirish Keskar, Richard Socher
State-of-the-art results on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et al. (2017) propose a…
Balancing Communication and Computation in Distributed Optimization
Albert S. Berahas, Raghu Bollapragada, Nitish Shirish Keskar +1
Methods for distributed optimization have received significant attention in recent years owing to their wide applicability in various domains. A distributed optimization method typ…
Regularizing and Optimizing LSTM Language Models
Stephen Merity, Nitish Shirish Keskar, Richard Socher
Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine transla…