28 citations · 73 across the 28 of their papers we have counts for
26 papers · 1 filter
Vocabulary Customization for Efficient Domain-Specific LLM Deployment
Christian Herold, Michael Kozielski, Nicholas Santavas +2
When using an LLM to process text outside the training domain(s), an often overlooked factor is vocabulary mismatch, where the general-domain tokenizer fails to capture frequent do…
Unilogit: Robust Machine Unlearning for LLMs Using Uniform-Target Self-Distillation
Stefan Vasilev, Christian Herold, Baohao Liao +3
This paper introduces Unilogit, a novel self-distillation method for machine unlearning in Large Language Models. Unilogit addresses the challenge of selectively forgetting specifi…
CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment
Jiangnan Li, Thuy-Trang Vu, Christian Herold +3
Naive joint training of large language models (LLMs) for multilingual preference alignment can suffer from negative interference. This is a known issue in multilingual training, wh…
ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning
Baohao Liao, Christian Herold, Seyyed Hadi Hashemi +3
As large language models (LLMs) scale, model compression is crucial for edge deployment and accessibility. Weight-only quantization reduces model size but suffers from performance…
Domain Adaptation of Foundation LLMs for e-Commerce
Christian Herold, Michael Kozielski, Tala Bazazo +6
We present the e-Llama models: 8 billion and 70 billion parameter large language models that are adapted towards the e-commerce domain. These models are meant as foundation models…
IKUN for WMT24 General MT Task: LLMs Are here for Multilingual Machine Translation
Baohao Liao, Christian Herold, Shahram Khadivi +1
This paper introduces two multilingual systems, IKUN and IKUN-C, developed for the general machine translation task in WMT24. IKUN and IKUN-C represent an open system and a constra…
LiLiuM: eBay's Large Language Models for e-commerce
Christian Herold, Michael Kozielski, Leonid Ekimov +3
We introduce the LiLiuM series of large language models (LLMs): 1B, 7B, and 13B parameter models developed 100% in-house to fit eBay's specific needs in the e-commerce domain. This…
Document-Level Language Models for Machine Translation
Frithjof Petrick, Christian Herold, Pavel Petrushkov +2
Despite the known limitations, most machine translation systems today still operate on the sentence-level. One reason for this is, that most parallel training data is only sentence…
Can Domains Be Transferred Across Languages in Multi-Domain Multilingual Neural Machine Translation?
Thuy-Trang Vu, Shahram Khadivi, Xuanli He +2
Previous works mostly focus on either multilingual or multi-domain aspects of neural machine translation (NMT). This paper investigates whether the domain information can be transf…
Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking
Iñigo Urteaga, Moulay-Zaïdane Draïdia, Tomer Lancewicki +1
We design and evaluate a Bayesian optimization framework for resource efficient pre-training of Transformer-based language models (TLMs). TLM pre-training requires high computation…
A Preordered RNN Layer Boosts Neural Machine Translation in Low Resource Settings
Mohaddeseh Bastan, Shahram Khadivi
Neural Machine Translation (NMT) models are strong enough to convey semantic and syntactic information from the source language to the target language. However, these models are su…
Towards Reinforcement Learning for Pivot-based Neural Machine Translation with Non-autoregressive Transformer
Evgeniia Tokarchuk, Jan Rosendahl, Weiyue Wang +4
Pivot-based neural machine translation (NMT) is commonly used in low-resource setups, especially for translation between non-English language pairs. It benefits from using high res…
Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer
Evgeniia Tokarchuk, Jan Rosendahl, Weiyue Wang +4
Complex natural language applications such as speech translation or pivot translation traditionally rely on cascaded models. However, cascaded models are known to be prone to error…
Back-translation for Large-Scale Multilingual Machine Translation
Baohao Liao, Shahram Khadivi, Sanjika Hewavitharana
This paper illustrates our approach to the shared task on large-scale multilingual machine translation in the sixth conference on machine translation (WMT-21). This work aims to bu…
Diving Deep into Context-Aware Neural Machine Translation
Jingjing Huo, Christian Herold, Yingbo Gao +3
Context-aware neural machine translation (NMT) is a promising direction to improve the translation quality by making use of the additional context, e.g., document-level translation…
Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages
Yunsu Kim, Petre Petrov, Pavel Petrushkov +2
We present effective pre-training strategies for neural machine translation (NMT) using parallel corpora involving a pivot language, i.e., source-pivot and pivot-target, leading to…
Generalizing Back-Translation in Neural Machine Translation
Miguel Graça, Yunsu Kim, Julian Schamper +2
Back-translation - data augmentation by translating target monolingual data - is a crucial component in modern neural machine translation (NMT). In this work, we reformulate back-t…
Word-based Domain Adaptation for Neural Machine Translation
Shen Yan, Leonard Dahlmann, Pavel Petrushkov +2
In this paper, we empirically investigate applying word-level weights to adapt neural machine translation to e-commerce domains, where small e-commerce datasets and large out-of-do…
Learning Bilingual Sentence Embeddings via Autoencoding and Computing Similarities with a Multilayer Perceptron
Yunsu Kim, Hendrik Rosendahl, Nick Rossenbach +3
We propose a novel model architecture and training algorithm to learn bilingual sentence embeddings from a combination of parallel and monolingual data. Our method connects autoenc…
Learning from Chunk-based Feedback in Neural Machine Translation
Pavel Petrushkov, Shahram Khadivi, Evgeny Matusov
We empirically investigate learning from partial feedback in neural machine translation (NMT), when partial feedback is collected by asking users to highlight a correct chunk of a…
Can Neural Machine Translation be Improved with User Feedback?
Julia Kreutzer, Shahram Khadivi, Evgeny Matusov +1
We present the first real-world application of methods for improving neural machine translation (NMT) with human reinforcement, based on explicit and implicit user feedback collect…
Neural Machine Translation Leveraging Phrase-based Models in a Hybrid Search
Leonard Dahlmann, Evgeny Matusov, Pavel Petrushkov +1
In this paper, we introduce a hybrid search for attention-based neural machine translation (NMT). A target phrase learned with statistical MT models extends a hypothesis in the NMT…
Neural and Statistical Methods for Leveraging Meta-information in Machine Translation
Shahram Khadivi, Patrick Wilken, Leonard Dahlmann +1
In this paper, we discuss different methods which use meta information and richer context that may accompany source language input to improve machine translation quality. We focus…
Graph-Based Semi-Supervised Conditional Random Fields For Spoken Language Understanding Using Unaligned Data
Mohammad Aliannejadi, Masoud Kiaeeha, Shahram Khadivi +1
We experiment graph-based Semi-Supervised Learning (SSL) of Conditional Random Fields (CRF) for the application of Spoken Language Understanding (SLU) on unaligned data. The aligne…