Simple and Effective Input Reformulations for Translation

Abstract

Foundation language models learn from their finetuning input context in different ways. In this paper, we reformulate inputs during finetuning for challenging translation tasks, leveraging model strengths from pretraining in novel ways to improve downstream performance. These reformulations are simple data level modifications, require no additional collection of training data or modification of data at inference time. They can be applied either on single language pair translation tasks or massively multilingual translation tasks. Experiments with these techniques demonstrate significant performance improvements up to $\text{3.5 chrF++ on the Flores200 translation benchmark}$. We hope our research accessibly improves finetuning data efficiency, enabling more effective training to scalably improve state-of-the-art performance. Our code is released $\text{href}.$

arXiv comment13 pages, 6 figures. To be published in Empirical Methods in Natural Language Processing (Main) 2023

References in corpus (26)

• Decoupled Weight Decay Regularization
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• PaLM: Scaling Language Modeling with Pathways
• Cross-lingual Language Model Pretraining
• Scaling Instruction-Finetuned Language Models
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• Learning to Reweight Examples for Robust Deep Learning
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• ERNIE: Enhanced Representation through Knowledge Integration
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Training Compute-Optimal Large Language Models
• mT5: A massively multilingual pre-trained text-to-text transformer
• Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting
• No Language Left Behind: Scaling Human-Centered Machine Translation
• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
• Large Language Models Are Human-Level Prompt Engineers
• Automatic Chain of Thought Prompting in Large Language Models
• What Makes Good In-Context Examples for GPT-$3$?
• Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
• An Empirical Exploration of Curriculum Learning for Neural Machine Translation
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
• The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
• MetaICL: Learning to Learn In Context