Publications (36)
Recovering document annotations for sentence-level bitext
Rachel Wicks, Matt Post, Philipp Koehn
Data availability limits the scope of any given task. In machine translation, historical models were incapable of handling longer contexts, so the lack of document-level datasets w…
The Multilingual TEDx Corpus for Speech Recognition and Translation
Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman +5
We present the Multilingual TEDx corpus, built to support speech recognition (ASR) and speech translation (ST) research across many non-English source languages. The corpus is a co…
Do GPTs Produce Less Literal Translations?
Vikas Raunak, Arul Menezes, Matt Post +1
Large Language Models (LLMs) such as GPT-3 have emerged as general-purpose language models capable of addressing many natural language generation or understanding tasks. On the tas…
Levenshtein Training for Word-level Quality Estimation
Shuoyang Ding, Marcin Junczys-Dowmunt, Matt Post +1
We propose a novel scheme to use the Levenshtein Transformer to perform the task of word-level quality estimation. A Levenshtein Transformer is a natural fit for this task: trained…
ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translation
J. Edward Hu, Rachel Rudinger, Matt Post +1
We present ParaBank, a large-scale English paraphrase dataset that surpasses prior work in both quantity and quality. Following the approach of ParaNMT, we train a Czech-English ne…
Escaping the sentence-level paradigm in machine translation
Matt Post, Marcin Junczys-Dowmunt
It is well-known that document context is vital for resolving a range of translation ambiguities, and in fact the document setting is the most natural setting for nearly all transl…
Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer
Elizabeth Salesky, Neha Verma, Philipp Koehn +1
We introduce and demonstrate how to effectively train multilingual machine translation models with pixel representations. We experiment with two different data settings with a vari…
Simulated Multiple Reference Training Improves Low-Resource Machine Translation
Huda Khayrallah, Brian Thompson, Matt Post +1
Many valid translations exist for a given sentence, yet machine translation (MT) is trained with a single reference translation, exacerbating data sparsity in low-resource settings…
A Study in Improving BLEU Reference Coverage with Diverse Automatic Paraphrasing
Rachel Bawden, Biao Zhang, Lisa Yankovskaya +2
We investigate a long-perceived shortcoming in the typical use of BLEU: its reliance on a single reference. Using modern neural paraphrasing techniques, we study whether automatica…
Robsut Wrod Reocginiton via semi-Character Recurrent Neural Network
Keisuke Sakaguchi, Kevin Duh, Matt Post +1
Language processing mechanism by humans is generally more robust than computers. The Cmabrigde Uinervtisy (Cambridge University) effect from the psycholinguistics literature has de…
Large-Scale Streaming End-to-End Speech Translation with Neural Transducers
Jian Xue, Peidong Wang, Jinyu Li +2
Neural transducers have been widely used in automatic speech recognition (ASR). In this paper, we introduce it to streaming end-to-end speech translation (ST), which aims to conver…
Operationalizing Specifications, In Addition to Test Sets for Evaluating Constrained Generative Models
Vikas Raunak, Matt Post, Arul Menezes
In this work, we present some recommendations on the evaluation of state-of-the-art generative models for constrained generation tasks. The progress on generative models has been r…
SALTED: A Framework for SAlient Long-Tail Translation Error Detection
Vikas Raunak, Matt Post, Arul Menezes
Traditional machine translation (MT) metrics provide an average measure of translation quality that is insensitive to the long tail of behavioral problems in MT. Examples include t…
Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversity
Brian Thompson, Matt Post
Recent work has shown that a multilingual neural machine translation (NMT) model can be used to judge how well a sentence paraphrases another sentence in the same language (Thompso…
Sockeye: A Toolkit for Neural Machine Translation
Felix Hieber, Tobias Domhan, Michael Denkowski +4
We describe Sockeye (version 1.12), an open-source sequence-to-sequence toolkit for Neural Machine Translation (NMT). Sockeye is a production-ready framework for training and apply…
Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation
Matt Post, David Vilar
The end-to-end nature of neural machine translation (NMT) removes many ways of manually guiding the translation process that were available in older paradigms. Recent work, however…
PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation
Lorenzo Proietti, Roman Grundkiewicz, Matt Post
We present PEAR (Pairwise Evaluation for Automatic Relative Scoring), a supervised quality estimation (QE) metric family that reframes reference-free machine translation (MT) evalu…
Improving Word Sense Disambiguation in Neural Machine Translation with Salient Document Context
Elijah Rippeth, Marine Carpuat, Kevin Duh +1
Lexical ambiguity is a challenging and pervasive problem in machine translation (\mt). We introduce a simple and scalable approach to resolve translation ambiguity by incorporating…
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?
Sorami Hisamoto, Matt Post, Kevin Duh
Data privacy is an important issue for "machine learning as a service" providers. We focus on the problem of membership inference attacks: given a data sample and black-box access…
Robust Open-Vocabulary Translation from Visual Text Representations
Elizabeth Salesky, David Etter, Matt Post
Machine translation models have discrete vocabularies and commonly use subword segmentation techniques to achieve an 'open vocabulary.' This approach relies on consistent and corre…
GLEU Without Tuning
Courtney Napoles, Keisuke Sakaguchi, Matt Post +1
The GLEU metric was proposed for evaluating grammatical error corrections using n-gram overlap with a set of reference sentences, as opposed to precision/recall of specific annotat…
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing
Brian Thompson, Matt Post
We frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser, conditioned on a human reference. We propo…
Token-level Ensembling of Models with Different Vocabularies
Rachel Wicks, Kartik Ravisankar, Xinchen Yang +2
Model ensembling is a technique to combine the predicted distributions of two or more models, often leading to improved robustness and performance. For ensembling in text generatio…
PyMarian: Fast Neural Machine Translation and Evaluation in Python
Thamme Gowda, Roman Grundkiewicz, Elijah Rippeth +2
The deep learning language of choice these days is Python; measured by factors such as available libraries and technical support, it is hard to beat. At the same time, software wri…
A Call for Clarity in Reporting BLEU Scores
Matt Post
The field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric. Although people refer to "the" BLEU…
CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translation
Rui Zhao, Jinyu Li, Ruchao Fan +1
Models for streaming speech translation (ST) can achieve high accuracy and low latency if they're developed with vast amounts of paired audio in the source language and written tex…
The JHU-Microsoft Submission for WMT21 Quality Estimation Shared Task
Shuoyang Ding, Marcin Junczys-Dowmunt, Matt Post +2
This paper presents the JHU-Microsoft joint submission for WMT 2021 quality estimation shared task. We only participate in Task 2 (post-editing effort estimation) of the shared tas…
Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies
Tom Kocmi, Vilém Zouhar, Christian Federmann +1
Ten years ago a single metric, BLEU, governed progress in machine translation research. For better or worse, there is no such consensus today, and consequently it is difficult for…
Identifying Context-Dependent Translations for Evaluation Set Production
Rachel Wicks, Matt Post
A major impediment to the transition to context-aware machine translation is the absence of good evaluation metrics and test sets. Sentences that require context to be translated c…
A Discriminative Neural Model for Cross-Lingual Word Alignment
Elias Stengel-Eskin, Tzu-Ray Su, Matt Post +1
We introduce a novel discriminative word alignment model, which we integrate into a Transformer-based machine translation model. In experiments based on a small number of labeled e…
Using of heterogeneous corpora for training of an ASR system
Jan Trmal, Gaurav Kumar, Vimal Manohar +3
The paper summarizes the development of the LVCSR system built as a part of the Pashto speech-translation system at the SCALE (Summer Camp for Applied Language Exploration) 2015 wo…
SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window
Vikas Raunak, Tom Kocmi, Matt Post
Reference-based metrics that operate at the sentence-level typically outperform quality estimation metrics, which have access only to the source and system output. This is unsurpri…
SOTASTREAM: A Streaming Approach to Machine Translation Training
Matt Post, Thamme Gowda, Roman Grundkiewicz +3
Many machine translation toolkits make use of a data preparation step wherein raw data is transformed into a tensor format that can be used directly by the trainer. This preparatio…
Additive Interventions Yield Robust Multi-Domain Machine Translation Models
Elijah Rippeth, Matt Post
Additive interventions are a recently-proposed mechanism for controlling target-side attributes in neural machine translation. In contrast to tag-based approaches which manipulate…
Dynamically Allocating Evaluation Effort for Model Ranking
Vilém Zouhar, Vilém Zouhar, Julia Kreutzer +6
While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation pr…
Grammatical Error Correction with Neural Reinforcement Learning
Keisuke Sakaguchi, Matt Post, Benjamin Van Durme
We propose a neural encoder-decoder model with reinforcement learning (NRL) for grammatical error correction (GEC). Unlike conventional maximum likelihood estimation (MLE), the mod…