Nearest Neighbor Machine Translation

Abstract

We introduce $k$-nearest-neighbor machine translation ($k$NN-MT), which predicts tokens with a nearest neighbor classifier over a large datastore of cached examples, using representations from a neural translation model for similarity search. This approach requires no additional training and scales to give the decoder direct access to billions of examples at test time, resulting in a highly expressive model that consistently improves performance across many settings. Simply adding nearest neighbor search improves a state-of-the-art German-English translation model by 1.5 BLEU. $k$NN-MT allows a single model to be adapted to diverse domains by using a domain-specific datastore, improving results by an average of 9.2 BLEU over zero-shot transfer, and achieving new state-of-the-art results -- without training on these domains. A massively multilingual model can also be specialized for particular language pairs, with improvements of 3 BLEU for translating from English into German and Chinese. Qualitatively, $k$NN-MT is easily interpretable; it combines source and target context to retrieve highly relevant examples.

arXiv commentICLR 2021

References in corpus (10)

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• REALM: Retrieval-Augmented Language Model Pre-Training
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• A Retrieve-and-Edit Framework for Predicting Structured Outputs
• Improving Neural Language Models with a Continuous Cache
• Guiding Neural Machine Translation with Retrieved Translation Pieces
• Augmenting Transformers with KNN-Based Composite Memory for Dialogue
• Retrieve and Refine: Improved Sequence Generation Models For Dialogue

Cited by in corpus (7)

• BASE Layers: Simplifying Training of Large, Sparse Models
• Personalized Federated Learning through Local Memorization
• GNN-LM: Language Modeling based on Global Contexts via GNN
• Neural Machine Translation with Monolingual Translation Memory
• Learning Sparse Prototypes for Text Generation
• Ember: No-Code Context Enrichment via Similarity-Based Keyless Joins
• Instance-Based Neural Dependency Parsing