7.4k citations · 28.3k across the 205 of their papers we have counts for
19 papers · 1 filter
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov +1
We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is lim…
Learning Markov Chain in Unordered Dataset
Yao-Hung Hubert Tsai, Han Zhao, Ruslan Salakhutdinov +1
The assumption that data samples are independently identically distributed is the backbone of many learning algorithms. Nevertheless, datasets often exhibit rich structure in pract…
Improving One-Shot Learning through Fusing Side Information
Yao-Hung Hubert Tsai, Ruslan Salakhutdinov
Deep Neural Networks (DNNs) often struggle with one-shot learning where we have only one or a few labeled training examples per category. In this paper, we argue that by using side…
A Generic Approach for Escaping Saddle points
Sashank J Reddi, Manzil Zaheer, Suvrit Sra +4
A central challenge to using first-order methods for optimizing nonconvex problems is the presence of saddle points. First-order methods often get stuck at saddle points, greatly d…
Block-Normalized Gradient Method: An Empirical Study for Training Deep Neural Network
Adams Wei Yu, Lei Huang, Qihang Lin +2
In this paper, we propose a generic and simple strategy for utilizing stochastic gradient information in optimization. The technique essentially contains two consecutive steps in e…
Gated-Attention Architectures for Task-Oriented Language Grounding
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi +2
To perform tasks specified by natural language instructions, autonomous agents need to extract semantically meaningful representations of language and map it to visual elements and…
On Unifying Deep Generative Models
Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov +1
Deep generative models have achieved impressive success in recent years. Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), as emerging families for genera…
Geometry of Optimization and Implicit Regularization in Deep Learning
Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov +1
We argue that the optimization plays a crucial role in generalization of deep learning models through implicit regularization. We do this by demonstrating that generalization abili…
Good Semi-supervised Learning that Requires a Bad GAN
Zihang Dai, Zhilin Yang, Fan Yang +2
Semi-supervised learning methods based on generative adversarial networks (GANs) obtained strong empirical results, but it is not clear 1) how the discriminator benefits from joint…
Question Answering from Unstructured Text by Retrieval and Comprehension
Yusuke Watanabe, Bhuwan Dhingra, Ruslan Salakhutdinov
Open domain Question Answering (QA) systems must interact with external knowledge sources, such as web pages, to find relevant information. Information sources like Wikipedia, howe…
Learning Robust Visual-Semantic Embeddings
Yao-Hung Hubert Tsai, Liang-Kang Huang, Ruslan Salakhutdinov
Many of the existing methods for learning joint embedding of images and text use only supervised information from paired images and its textual attributes. Taking advantage of the…
Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
Zhilin Yang, Ruslan Salakhutdinov, William W. Cohen
Recent papers have shown that neural networks obtain state-of-the-art performance on several different sequence tagging tasks. One appealing property of such systems is their gener…
Linguistic Knowledge as Memory for Recurrent Neural Networks
Bhuwan Dhingra, Zhilin Yang, William W. Cohen +1
Training recurrent neural networks to model long term dependencies is difficult. Hence, we propose to use external linguistic knowledge as an explicit signal to inform the model wh…
A Comparative Study of Word Embeddings for Reading Comprehension
Bhuwan Dhingra, Hanxiao Liu, Ruslan Salakhutdinov +1
The focus of past machine learning research for Reading Comprehension tasks has been primarily on the design of novel deep learning architectures. Here we show that seemingly minor…
Deep Sets
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh +3
We study the problem of designing models for machine learning tasks defined on \emph{sets}. In contrast to traditional approach of operating on fixed dimensional vectors, we consid…
Toward Controlled Generation of Text
Zhiting Hu, Zichao Yang, Xiaodan Liang +2
Generic generation and manipulation of text is challenging and has limited success compared to recent deep generative modeling in visual domain. This paper aims at generating plaus…
Neural Map: Structured Memory for Deep Reinforcement Learning
Emilio Parisotto, Ruslan Salakhutdinov
A critical component to enabling intelligent reasoning in partially observable environments is memory. Despite this importance, Deep Reinforcement Learning (DRL) agents have so far…
Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov +1
Recent work on generative modeling of text has found that variational auto-encoders (VAE) incorporating LSTM decoders perform worse than simpler LSTM language models (Bowman et al.…
Semi-Supervised QA with Generative Domain-Adaptive Nets
Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov +1
We study the problem of semi-supervised question answering----utilizing unlabeled text to boost the performance of question answering models. We propose a novel training framework,…