70 citations · 114 across the 13 of their papers we have counts for
16 papers
Information Flow Routes: Automatically Interpreting Language Models at Scale
Javier Ferrando, Elena Voita
Information flows by routes inside the network via mechanisms implemented in the model. These routes can be represented as graphs where nodes correspond to token representations an…
LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models
Igor Tufanov, Karen Hambardzumyan, Javier Ferrando +1
We present the LM Transparency Tool (LM-TT), an open-source interactive toolkit for analyzing the internal workings of Transformer-based language models. Differently from previousl…
Neurons in Large Language Models: Dead, N-gram, Positional
Elena Voita, Javier Ferrando, Christoforos Nalmpantis
We analyze a family of large language models in such a lightweight manner that can be done on a single GPU. Specifically, we focus on the OPT family of models ranging from 125m to…
HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation
David Dale, Elena Voita, Janice Lam +6
Hallucinations in machine translation are translations that contain information completely unrelated to the input. Omissions are translations that do not include some of the input…
Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better
David Dale, Elena Voita, Loïc Barrault +1
While the problem of hallucinations in neural machine translation has long been recognized, so far the progress on its alleviation is very little. Indeed, recently it turned out th…
Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation
Nuno M. Guerreiro, Elena Voita, André F. T. Martins
Although the problem of hallucinations in neural machine translation (NMT) has received some attention, research on this highly pathological phenomenon lacks solid ground. Previous…
Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT
Elena Voita, Rico Sennrich, Ivan Titov
Differently from the traditional statistical MT that decomposes the translation task into distinct separately learned components, neural machine translation uses a single neural ne…
Embedding Words in Non-Vector Space with Unsupervised Graph Learning
Max Ryabinin, Sergei Popov, Liudmila Prokhorenkova +1
It has become a de-facto standard to represent words as elements of a vector space (word2vec, GloVe). While this approach is convenient, it is unnatural for language: words form a…
Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation
Elena Voita, Rico Sennrich, Ivan Titov
In Neural Machine Translation (and, more generally, conditional language modeling), the generation of a target token is influenced by two types of context: the source and the prefi…
Information-Theoretic Probing with Minimum Description Length
Elena Voita, Ivan Titov
To measure how well pretrained representations encode some linguistic property, it is common to use accuracy of a probe, i.e. a classifier trained to predict the property from the…
Sequence Modeling with Unconstrained Generation Order
Dmitrii Emelianenko, Elena Voita, Pavel Serdyukov
The dominant approach to sequence generation is to produce a sequence in some predefined order, e.g. left to right. In contrast, we propose a more general model that can generate t…
Context-Aware Monolingual Repair for Neural Machine Translation
Elena Voita, Rico Sennrich, Ivan Titov
Modern sentence-level NMT systems often produce plausible translations of isolated sentences. However, when put in context, these translations may end up being inconsistent with ea…
BPE-Dropout: Simple and Effective Subword Regularization
Ivan Provilkov, Dmitrii Emelianenko, Elena Voita
Subword segmentation is widely used to address the open vocabulary problem in machine translation. The dominant approach to subword segmentation is Byte Pair Encoding (BPE), which…
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
Elena Voita, Rico Sennrich, Ivan Titov
We seek to understand how the representations of individual tokens and the structure of the learned feature space evolve between layers in deep neural networks under different lear…
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Elena Voita, David Talbot, Fedor Moiseev +2
Multi-head self-attention is a key component of the Transformer, a state-of-the-art architecture for neural machine translation. In this work we evaluate the contribution made by i…
Context-Aware Neural Machine Translation Learns Anaphora Resolution
Elena Voita, Pavel Serdyukov, Rico Sennrich +1
Standard machine translation systems process sentences in isolation and hence ignore extra-sentential information, even though extended context can both prevent mistakes in ambiguo…