565 citations · 892 across the 17 of their papers we have counts for
22 papers
Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation
Verna Dankers, Vikas Raunak
In this work, we explore how instance-level memorization in the teacher Neural Machine Translation (NMT) model gets inherited by the student model in sequence-level knowledge disti…
Do GPTs Produce Less Literal Translations?
Vikas Raunak, Arul Menezes, Matt Post +1
Large Language Models (LLMs) such as GPT-3 have emerged as general-purpose language models capable of addressing many natural language generation or understanding tasks. On the tas…
Leveraging GPT-4 for Automatic Translation Post-Editing
Vikas Raunak, Amr Sharaf, Yiren Wang +2
While Neural Machine Translation (NMT) represents the leading approach to Machine Translation (MT), the outputs of NMT models still require translation post-editing to rectify erro…
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf +6
Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoro…
Rank-One Editing of Encoder-Decoder Models
Vikas Raunak, Arul Menezes
Large sequence to sequence models for tasks such as Neural Machine Translation (NMT) are usually trained over hundreds of millions of samples. However, training is just the origin…
Operationalizing Specifications, In Addition to Test Sets for Evaluating Constrained Generative Models
Vikas Raunak, Matt Post, Arul Menezes
In this work, we present some recommendations on the evaluation of state-of-the-art generative models for constrained generation tasks. The progress on generative models has been r…
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop, :, Teven Le Scao +391
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to wi…
Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks
Vikas Raunak, Arul Menezes
Memorization presents a challenge for several constrained Natural Language Generation (NLG) tasks such as Neural Machine Translation (NMT), wherein the proclivity of neural models…
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran +74
Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…
SALTED: A Framework for SAlient Long-Tail Translation Error Detection
Vikas Raunak, Matt Post, Arul Menezes
Traditional machine translation (MT) metrics provide an average measure of translation quality that is insensitive to the long tail of behavioral problems in MT. Examples include t…
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation
Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann +122
Data augmentation is an important component in the robustness evaluation of models in natural language processing (NLP) and in enhancing the diversity of the data they are trained…
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks
Siddharth Dalmia, Brian Yan, Vikas Raunak +2
End-to-end approaches for sequence tasks are becoming increasingly popular. Yet for complex sequence tasks, like speech translation, systems that cascade several models trained on…
The Curious Case of Hallucinations in Neural Machine Translation
Vikas Raunak, Arul Menezes, Marcin Junczys-Dowmunt
In this work, we study hallucinations in Neural Machine Translation (NMT), which lie at an extreme end on the spectrum of NMT pathologies. Firstly, we connect the phenomenon of hal…
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal +53
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of auto…
On Long-Tailed Phenomena in Neural Machine Translation
Vikas Raunak, Siddharth Dalmia, Vivek Gupta +1
State-of-the-art Neural Machine Translation (NMT) models struggle with generating low-frequency tokens, tackling which remains a major challenge. The analysis of long-tailed phenom…
Ranking Clarification Questions via Natural Language Inference
Vaibhav Kumar, Vikas Raunak, Jamie Callan
Given a natural language query, teaching machines to ask clarifying questions is of immense utility in practical natural language processing systems. Such interactions could help i…
On Compositionality in Neural Machine Translation
Vikas Raunak, Vaibhav Kumar, Florian Metze
We investigate two specific manifestations of compositionality in Neural Machine Translation (NMT) : (1) Productivity - the ability of the model to extend its predictions beyond th…
On Leveraging the Visual Modality for Neural Machine Translation
Vikas Raunak, Sang Keun Choe, Quanyang Lu +2
Leveraging the visual modality effectively for Neural Machine Translation (NMT) remains an open problem in computational linguistics. Recently, Caglayan et al. posit that the obser…
On Dimensional Linguistic Properties of the Word Embedding Space
Vikas Raunak, Vaibhav Kumar, Vivek Gupta +1
Word embeddings have become a staple of several natural language processing tasks, yet much remains to be understood about their properties. In this work, we analyze word embedding…
Learned In Speech Recognition: Contextual Acoustic Word Embeddings
Shruti Palaskar, Vikas Raunak, Florian Metze
End-to-end acoustic-to-word speech recognition models have recently gained popularity because they are easy to train, scale well to large amounts of training data, and do not requi…
Simple and Effective Dimensionality Reduction for Word Embeddings
Vikas Raunak
Word embeddings have become the basic building blocks for several natural language processing and information retrieval tasks. Pre-trained word embeddings are used in several downs…