105 citations · 147 across the 17 of their papers we have counts for
19 papers
MegaWika: Millions of reports and their sources across 50 diverse languages
Samuel Barham, Orion Weller, Michelle Yuan +9
To foster the development of new models for collaborative AI-assisted report generation, we introduce MegaWika, consisting of 13 million Wikipedia articles in 50 diverse languages,…
Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution
Tianjian Li, Kenton Murray
Zero-shot cross-lingual transfer is when a multilingual model is trained to perform a task in one language and then is applied to another language. Although the zero-shot cross-lin…
Exploring Representational Disparities Between Multilingual and Bilingual Translation Models
Neha Verma, Kenton Murray, Kevin Duh
Multilingual machine translation has proven immensely useful for both parameter efficiency and overall performance across many language pairs via complete multilingual parameter sh…
Condensing Multilingual Knowledge with Lightweight Language-Specific Modules
Haoran Xu, Weiting Tan, Shuyue Stella Li +4
Incorporating language-specific (LS) modules is a proven method to boost performance in multilingual machine translation. This approach bears similarity to Mixture-of-Experts (MoE)…
Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity
Haoran Xu, Maha Elbayad, Kenton Murray +2
Mixture-of-experts (MoE) models that employ sparse activation have demonstrated effectiveness in significantly increasing the number of parameters while maintaining low computation…
Language Agnostic Code-Mixing Data Augmentation by Predicting Linguistic Patterns
Shuyue Stella Li, Kenton Murray
In this work, we focus on intrasentential code-mixing and propose several different Synthetic Code-Mixing (SCM) data augmentation methods that outperform the baseline on downstream…
The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains
Haoran Xu, Philipp Koehn, Kenton Murray
Recent model pruning methods have demonstrated the ability to remove redundant parameters without sacrificing model performance. Common methods remove redundant parameters accordin…
Por Qué Não Utiliser Alla Språk? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer
Haoran Xu, Kenton Murray
The current state-of-the-art for few-shot cross-lingual transfer learning first trains on abundant labeled data in the source language and then fine-tunes with a few examples on th…
Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models
Suraj Nair, Eugene Yang, Dawn Lawrie +5
The advent of transformer-based models such as BERT has led to the rise of neural ranking models. These models have improved the effectiveness of retrieval systems well beyond that…
BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
Haoran Xu, Benjamin Van Durme, Kenton Murray
The success of bidirectional encoders using masked language models, such as BERT, on numerous natural language processing tasks has prompted researchers to attempt to incorporate t…
Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solution
Toan Q. Nguyen, Kenton Murray, David Chiang
In this paper, we investigate the driving factors behind concatenation, a simple but effective data augmentation method for low-resource neural machine translation. Our experiments…
Joint Universal Syntactic and Semantic Parsing
Elias Stengel-Eskin, Kenton Murray, Sheng Zhang +2
While numerous attempts have been made to jointly parse syntax and semantics, high performance in one domain typically comes at the price of performance in the other. This trade-of…
Gradual Fine-Tuning for Low-Resource Domain Adaptation
Haoran Xu, Seth Ebner, Mahsa Yarmohammadi +3
Fine-tuning is known to improve NLP models by adapting an initial model trained on more plentiful but less domain-salient examples to data in a target domain. Such domain adaptatio…
Efficiency through Auto-Sizing: Notre Dame NLP's Submission to the WNGT 2019 Efficiency Task
Kenton Murray, Brian DuSell, David Chiang
This paper describes the Notre Dame Natural Language Processing Group's (NDNLP) submission to the WNGT 2019 shared task (Hayashi et al., 2019). We investigated the impact of auto-s…
Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation
Kenton Murray, Jeffery Kinnison, Toan Q. Nguyen +2
Neural sequence-to-sequence models, particularly the Transformer, are the state of the art in machine translation. Yet these neural networks are very sensitive to architecture and…
An Empirical Exploration of Curriculum Learning for Neural Machine Translation
Xuan Zhang, Gaurav Kumar, Huda Khayrallah +6
Machine translation systems based on deep neural networks are expensive to train. Curriculum learning aims to address this issue by choosing the order in which samples are presente…
Correcting Length Bias in Neural Machine Translation
Kenton Murray, David Chiang
We study two problems in neural machine translation (NMT). First, in beam search, whereas a wider beam should in principle help translation, it often hurts NMT. Second, NMT has a t…
Semantic Scan: Detecting Subtle, Spatially Localized Events in Text Streams
Abhinav Maurya, Kenton Murray, Yandong Liu +3
Early detection and precise characterization of emerging topics in text streams can be highly useful in applications such as timely and targeted public health interventions and dis…
Auto-Sizing Neural Networks: With Applications to n-gram Language Models
Kenton Murray, David Chiang
Neural networks have been shown to improve performance across a range of natural-language tasks. However, designing and training them can be complicated. Frequently, researchers re…