14.5k citations · 48.8k across the 185 of their papers we have counts for
39 papers · 1 filter
Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
Margaret Li, Stephen Roller, Ilia Kulikov +4
Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely to…
Multi-Stage Document Ranking with BERT
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho +1
The advent of deep neural networks pre-trained via language modeling tasks has spurred a number of successful applications in natural language processing. This work explores one su…
Generalized Inner Loop Meta-Learning
Edward Grefenstette, Brandon Amos, Denis Yarats +6
Many (but not all) approaches self-qualifying as "meta-learning" in deep learning and reinforcement learning fit a common pattern of approximating the solution to a nested optimiza…
Capacity, Bandwidth, and Compositionality in Emergent Language Learning
Cinjon Resnick, Abhinav Gupta, Jakob Foerster +2
Many recent works have discussed the propensity, or lack thereof, for emergent languages to exhibit properties of natural languages. A favorite in the literature is learning compos…
Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models
Tianxing He, Jun Liu, Kyunghyun Cho +4
In this work, we study how the finetuning stage in the pretrain-finetune framework changes the behavior of a pretrained neural language generator. We focus on the transformer encod…
Neural Text Generation with Unlikelihood Training
Sean Welleck, Ilia Kulikov, Stephen Roller +3
Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and de…
Inducing Constituency Trees through Neural Machine Translation
Phu Mon Htut, Kyunghyun Cho, Samuel R. Bowman
Latent tree learning(LTL) methods learn to parse sentences using only indirect supervision from a downstream task. Recent advances in latent tree learning have made it possible to…
Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set
Katharina Kann, Kyunghyun Cho, Samuel R. Bowman
Development sets are impractical to obtain for real low-resource languages, since using all available data for training is often more effective. However, development sets are widel…
Finding Generalizable Evidence by Learning to Convince Q&A Models
Ethan Perez, Siddharth Karamcheti, Rob Fergus +3
We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question-answering (QA) as a testbed. We train evidence agent…
Countering Language Drift via Visual Grounding
Jason Lee, Kyunghyun Cho, Douwe Kiela
Emergent multi-agent communication protocols are very different from natural language and not easily interpretable by humans. We find that agents that were initially pretrained to…
Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
Cheolhyoung Lee, Kyunghyun Cho, Wanmo Kang
In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabel…
Neural Machine Translation with Byte-Level Subwords
Changhan Wang, Kyunghyun Cho, Jiatao Gu
Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich la…
Dynamics-aware Embeddings
William Whitney, Rajat Agarwal, Kyunghyun Cho +1
In this paper we consider self-supervised representation learning to improve sample efficiency in reinforcement learning (RL). We propose a forward prediction objective for simulta…
Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
Raphael Shu, Jason Lee, Hideki Nakayama +1
Although neural machine translation models reached high translation quality, the autoregressive nature makes inference difficult to parallelize and leads to high translation latenc…
Improving localization-based approaches for breast cancer screening exam classification
Thibault Févry, Jason Phang, Nan Wu +4
We trained and evaluated a localization-based deep CNN for breast cancer screening exam classification on over 200,000 exams (over 1,000,000 images). Our model achieves an AUC of 0…
Screening Mammogram Classification with Prior Exams
Jungkyu Park, Jason Phang, Yiqiu Shen +5
Radiologists typically compare a patient's most recent breast cancer screening exam to their previous ones in making informed diagnoses. To reflect this practice, we propose new ne…
A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks
Owen Marschall, Kyunghyun Cho, Cristina Savin
We present a framework for compactly summarizing many recent results in efficient and/or biologically plausible online training of recurrent neural networks (RNN). The framework or…
Can Unconditional Language Models Recover Arbitrary Sentences?
Nishant Subramani, Samuel R. Bowman, Kyunghyun Cho
Neural network-based generative language models like ELMo and BERT can work effectively as general purpose sentence encoders in text classification without further fine-tuning. Is…
Deep Unsupervised Drum Transcription
Keunwoo Choi, Kyunghyun Cho
We introduce DrummerNet, a drum transcription system that is trained in an unsupervised manner. DrummerNet does not require any ground-truth transcription and, with the data-scalab…
Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
Jiatao Gu, Yong Wang, Kyunghyun Cho +1
Zero-shot translation, translating between language pairs on which a Neural Machine Translation (NMT) system has never been trained, is an emergent property when training the syste…
Globally-Aware Multiple Instance Classifier for Breast Cancer Screening
Yiqiu Shen, Nan Wu, Jason Phang +5
Deep learning models designed for visual classification tasks on natural images have become prevalent in medical image analysis. However, medical images differ from typical natural…
Multi-Turn Beam Search for Neural Dialogue Modeling
Ilia Kulikov, Jason Lee, Kyunghyun Cho
In neural dialogue modeling, a neural network is trained to predict the next utterance, and at inference time, an approximate decoding algorithm is used to generate next utterances…
Using local plasticity rules to train recurrent neural networks
Owen Marschall, Kyunghyun Cho, Cristina Savin
To learn useful dynamics on long time scales, neurons must use plasticity rules that account for long-term, circuit-wide effects of synaptic changes. In other words, neural circuit…
Automatic Machine Learning by Pipeline Synthesis using Model-Based Reinforcement Learning and a Grammar
Iddo Drori, Yamuna Krishnamurthy, Raoni Lourenco +4
Automatic machine learning is an important problem in the forefront of machine learning. The strongest AutoML systems are based on neural networks, evolutionary algorithms, and Bay…