Publications (25)
Fair Representation in Parliamentary Summaries: Measuring and Mitigating Inclusion Bias
Eoghan Cunningham, James Cross, Derek Greene
The The use of Large language models (LLMs) to summarise parliamentary proceedings presents a promising means of increasing the accessibility of democratic participation. However,…
Lifting the Curse of Multilinguality by Pre-training Modular Transformers
Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin +4
Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this i…
XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignment
Ahmed El-Kishky, Adithya Renduchintala, James Cross +2
Cross-lingual named-entity lexica are an important resource to multilingual NLP tasks such as machine translation and cross-lingual wikification. While knowledge bases contain a la…
Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders
Xiang Kong, Adithya Renduchintala, James Cross +3
Recent work in multilingual translation advances translation quality surpassing bilingual baselines using deep transformer models with increased capacity. However, the extra latenc…
Improving Zero-Shot Translation by Disentangling Positional Information
Danni Liu, Jan Niehues, James Cross +2
Multilingual neural machine translation has shown the capability of directly translating between language pairs unseen in training, i.e. zero-shot translation. Despite being concep…
Data Selection Curriculum for Neural Machine Translation
Tasnim Mohiuddin, Philipp Koehn, Vishrav Chaudhary +3
Neural Machine Translation (NMT) models are typically trained on heterogeneous data that are concatenated and randomly shuffled. However, not all of the training data are equally u…
Good, Better, Best: Choosing Word Embedding Context
James Cross, Bing Xiang, Bowen Zhou
We propose two methods of learning vector representations of words and phrases that each combine sentence context with structural features extracted from dependency trees. Using se…
Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation
Eoghan Cunningham, Derek Greene, James Cross +1
Understanding how policy is debated and justified in parliament is a fundamental aspect of the democratic process. However, the volume and complexity of such debates mean that outs…
No Language Left Behind: Scaling Human-Centered Machine Translation
NLLB Team, Marta R. Costa-jussÃ, James Cross +36
Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today. However,…
Tricks for Training Sparse Translation Models
Dheeru Dua, Shruti Bhosale, Vedanuj Goswami +3
Multi-task learning with an unbalanced data distribution skews model learning towards high resource tasks, especially when model capacity is fixed and fully shared across all tasks…
Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation
Lorcan McLaren, James Cross, Zuzanna Krakowska +2
Political scientists are rapidly adopting large language models (LLMs) for text annotation, yet the sensitivity of annotation results to implementation choices remains poorly under…
Alternative Input Signals Ease Transfer in Multilingual Machine Translation
Simeng Sun, Angela Fan, James Cross +4
Recent work in multilingual machine translation (MMT) has focused on the potential of positive transfer between languages, particularly cases where higher-resourced languages can b…
On the Evaluation of Machine Translation for Terminology Consistency
Md Mahfuz ibn Alam, Antonios Anastasopoulos, Laurent Besacier +4
As neural machine translation (NMT) systems become an important part of professional translator pipelines, a growing body of work focuses on combining NMT with terminologies. In ma…
Facebook AI WMT21 News Translation Task Submission
Chau Tran, Shruti Bhosale, James Cross +3
We describe Facebook's multilingual model submission to the WMT2021 shared task on news translation. We participate in 14 language directions: English to and from Czech, German, Ha…
Non-Autoregressive Machine Translation with Disentangled Context Transformer
Jungo Kasai, James Cross, Marjan Ghazvininejad +1
State-of-the-art neural machine translation models generate a translation from left to right and every step is conditioned on the previously generated tokens. The sequential nature…
Multilingual Machine Translation with Hyper-Adapters
Christos Baziotis, Mikel Artetxe, James Cross +1
Multilingual machine translation suffers from negative interference across languages. A common solution is to relax parameter sharing with language-specific modules like adapters.…
Learn to Talk via Proactive Knowledge Transfer
Qing Sun, James Cross
Knowledge Transfer has been applied in solving a wide variety of problems. For example, knowledge can be transferred between tasks (e.g., learning to handle novel situations by lev…
Monotonic Multihead Attention
Xutai Ma, Juan Pino, James Cross +2
Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approaches for this task either apply a fixe…
Span-Based Constituency Parsing with a Structure-Label System and Provably Optimal Dynamic Oracles
James Cross, Liang Huang
Parsing accuracy using efficient greedy transition systems has improved dramatically in recent years thanks to neural networks. Despite striking results in dependency parsing, howe…
Simple Fusion: Return of the Language Model
Felix Stahlberg, James Cross, Veselin Stoyanov
Neural Machine Translation (NMT) typically leverages monolingual data in training through backtranslation. We investigate an alternative simple method to use monolingual data for N…
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
Jungo Kasai, Nikolaos Pappas, Hao Peng +2
Much recent effort has been invested in non-autoregressive neural machine translation, which appears to be an efficient alternative to state-of-the-art autoregressive machine trans…
Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications
Shuo Sun, Ahmed El-Kishky, Vishrav Chaudhary +3
Sentence-level Quality estimation (QE) of machine translation is traditionally formulated as a regression task, and the performance of QE models is typically measured by Pearson co…
How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?
Shiyue Zhang, Vishrav Chaudhary, Naman Goyal +4
A multilingual tokenizer is a fundamental component of multilingual neural machine translation. It is trained from a multilingual corpus. Since a skewed data distribution is consid…
Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages
Simeng Sun, Maha Elbayad, Anna Sun +1
With multilingual machine translation (MMT) models continuing to grow in size and number of supported languages, it is natural to reuse and upgrade existing models to save computat…
Incremental Parsing with Minimal Features Using Bi-Directional LSTM
James Cross, Liang Huang
Recently, neural network approaches for parsing have largely automated the combination of individual features, but still rely on (often a larger number of) atomic features created…