Simple and Effective Input Reformulations for Translation

Abstract

Foundation language models learn from their finetuning input context in different ways. In this paper, we reformulate inputs during finetuning for challenging translation tasks, leveraging model strengths from pretraining in novel ways to improve downstream performance. These reformulations are simple data level modifications, require no additional collection of training data or modification of data at inference time. They can be applied either on single language pair translation tasks or massively multilingual translation tasks. Experiments with these techniques demonstrate significant performance improvements up to $\text{3.5 chrF++ on the Flores200 translation benchmark}$. We hope our research accessibly improves finetuning data efficiency, enabling more effective training to scalably improve state-of-the-art performance. Our code is released $\text{href}.$

13 pages, 6 figures. To be published in Empirical Methods in Natural Language Processing (Main) 2023

References in corpus (10)

• LLaMA: Open and Efficient Foundation Language Models
• Cross-lingual Language Model Pretraining
• Scaling Instruction-Finetuned Language Models
• ERNIE: Enhanced Representation through Knowledge Integration
• No Language Left Behind: Scaling Human-Centered Machine Translation
• Large Language Models Are Human-Level Prompt Engineers
• Automatic Chain of Thought Prompting in Large Language Models
• What Makes Good In-Context Examples for GPT-$3$?
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
• The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation