738 citations · 1.6k across the 7 of their papers we have counts for
7 papers
Symbolic Discovery of Optimization Algorithms
Xiangning Chen, Chen Liang, Da Huang +9
We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient sea…
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh +14
We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compos…
Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
Zhao Chen, Jiquan Ngiam, Yanping Huang +4
The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights. However, these…
Learning Longer-term Dependencies in RNNs with Auxiliary Losses
Trieu H. Trinh, Andrew M. Dai, Minh-Thang Luong +1
Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge. Most approaches use backpropaga…
Effective Approaches to Attention-based Neural Machine Translation
Minh-Thang Luong, Hieu Pham, Christopher D. Manning
An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there…
When Are Tree Structures Necessary for Deep Learning of Representations?
Jiwei Li, Minh-Thang Luong, Dan Jurafsky +1
Recursive neural models, which use syntactic parse trees to recursively generate representations bottom-up, are a popular architecture. But there have not been rigorous evaluations…
Addressing the Rare Word Problem in Neural Machine Translation
Minh-Thang Luong, Ilya Sutskever, Quoc V. Le +2
Neural Machine Translation (NMT) is a new approach to machine translation that has shown promising results that are comparable to traditional approaches. A significant weakness in…