Publications (25)
Efficient Methods for Natural Language Processing: A Survey
Marcos Treviso, Ji-Ung Lee, Tianchu Ji +19
Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance mea…
Exploring Diversity in Back Translation for Low-Resource Machine Translation
Laurie Burchell, Alexandra Birch, Kenneth Heafield
Back translation is one of the most widely used methods for improving the performance of neural machine translation systems. Recent research has sought to enhance the effectiveness…
The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020
Tobias Domhan, Michael Denkowski, David Vilar +3
We present Sockeye 2, a modernized and streamlined version of the Sockeye neural machine translation (NMT) toolkit. New features include a simplified code base through the use of M…
Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation
Nikolay Bogoychev, Marcin Junczys-Dowmunt, Kenneth Heafield +1
In order to extract the best possible performance from asynchronous stochastic gradient descent one must increase the mini-batch size and scale the learning rate accordingly. In or…
Making Asynchronous Stochastic Gradient Descent Work for Transformers
Alham Fikri Aji, Kenneth Heafield
Asynchronous stochastic gradient descent (SGD) is attractive from a speed perspective because workers do not wait for synchronization. However, the Transformer model converges poor…
Fully Synthetic Data Improves Neural Machine Translation with Knowledge Distillation
Alham Fikri Aji, Kenneth Heafield
This paper explores augmenting monolingual data for knowledge distillation in neural machine translation. Source language monolingual text can be incorporated as a forward translat…
Iterative Translation Refinement with Large Language Models
Pinzhen Chen, Zhicheng Guo, Barry Haddow +1
We propose iteratively prompting a large language model to self-correct a translation, with inspiration from their strong language understanding and translation capability as well…
Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca
Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev +3
Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. While such efforts are…
Neural Machine Translation with 4-Bit Precision and Beyond
Alham Fikri Aji, Kenneth Heafield
Neural Machine Translation (NMT) is resource intensive. We design a quantization procedure to compress NMT models better for devices with limited hardware capability. Because most…
The Galex Ultraviolet Variability (GUVV) Catalog
Barry Y. Welsh, Jonathan M. Wheatley, Kenneth Heafield +24
We present Version 1.0 of the NASA Galaxy Evolution Explorer (GALEX) ultraviolet variability catalog (GUVV) that contains information on 84 time-variable and transient sources gain…
TranslateLocally: Blazing-fast translation running on the local CPU
Nikolay Bogoychev, Jelmer Van der Linde, Kenneth Heafield
Every day, millions of people sacrifice their privacy and browsing habits in exchange for online machine translation. Companies and governments with confidentiality requirements of…
Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha +1
Previously, neural methods in grammatical error correction (GEC) did not reach state-of-the-art results compared to phrase-based statistical machine translation (SMT) baselines. We…
The University of Edinburgh's Neural MT Systems for WMT17
Rico Sennrich, Alexandra Birch, Anna Currey +5
This paper describes the University of Edinburgh's submissions to the WMT17 shared news translation and biomedical translation tasks. We participated in 12 translation directions f…
Code-Switched Language Identification is Harder Than You Think
Laurie Burchell, Alexandra Birch, Robert P. Thompson +1
Code switching (CS) is a very common phenomenon in written and spoken communication but one that is handled poorly by many natural language processing applications. Looking to the…
Marian: Fast Neural Machine Translation in C++
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak +9
We present Marian, an efficient and self-contained Neural Machine Translation framework with an integrated automatic differentiation engine based on dynamic computation graphs. Mar…
Sparse Communication for Distributed Gradient Descent
Alham Fikri Aji, Kenneth Heafield
We make distributed stochastic gradient descent faster by exchanging sparse updates instead of dense updates. Gradient updates are positively skewed as most updates are near zero,…
Exploring Hyper-Parameter Optimization for Neural Machine Translation on GPU Architectures
Robert Lim, Kenneth Heafield, Hieu Hoang +2
Neural machine translation (NMT) has been accelerated by deep learning neural networks over statistical-based approaches, due to the plethora and programmability of commodity heter…
Multi-Source Syntactic Neural Machine Translation
Anna Currey, Kenneth Heafield
We introduce a novel multi-source technique for incorporating source syntax into neural machine translation using linearized parses. This is achieved by employing separate encoders…
Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translation
Adithya Renduchintala, Denise Diaz, Kenneth Heafield +2
Is bias amplified when neural machine translation (NMT) models are optimized for speed and evaluated on generic test sets using BLEU? We investigate architectures and techniques co…
Marian: Cost-effective High-Quality Neural Machine Translation in C++
Marcin Junczys-Dowmunt, Kenneth Heafield, Hieu Hoang +2
This paper describes the submissions of the "Marian" team to the WNMT 2018 shared task. We investigate combinations of teacher-student training, low-precision matrix products, auto…
The Llama 3 Herd of Models
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri +556
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models th…
An Open Dataset and Model for Language Identification
Laurie Burchell, Alexandra Birch, Nikolay Bogoychev +1
Language identification (LID) is a fundamental step in many natural language processing pipelines. However, current LID systems are far from perfect, particularly on lower-resource…
Approaching Neural Chinese Word Segmentation as a Low-Resource Machine Translation Task
Pinzhen Chen, Kenneth Heafield
Chinese word segmentation has entered the deep learning era which greatly reduces the hassle of feature engineering. Recently, some researchers attempted to treat it as character-l…
Fast Neural Machine Translation Implementation
Hieu Hoang, Tomasz Dwojak, Rihards Krislauks +2
This paper describes the submissions to the efficiency track for GPUs at the Workshop for Neural Machine Translation and Generation by members of the University of Edinburgh, Adam…
No Language Left Behind: Scaling Human-Centered Machine Translation
NLLB Team, Marta R. Costa-jussÃ, James Cross +36
Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today. However,…