633 citations · 2.5k across the 52 of their papers we have counts for
55 papers
Cross-lingual Self-Consistency for Multilingual Reasoning with Language Models
Ahmed Elhady, Eneko Agirre, Mikel Artetxe
Despite expanding their multilingual coverage, the advanced reasoning capabilities of LLMs remain largely confined to a few high-resource languages like English. To address this, w…
Translate, then Detect: Leveraging Machine Translation for Cross-Lingual Toxicity Classification
Samuel J. Bell, Eduardo Sánchez, David Dale +3
Multilingual toxicity detection remains a significant challenge due to the scarcity of training data and resources for many languages. While prior work has leveraged the translate-…
Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque
Oscar Sainz, Naiara Perez, Julen Etxaniz +9
Instructing language models with user intent requires large instruction datasets, which are only available for a limited set of languages. In this paper, we explore alternatives to…
Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation
Ahmed Elhady, Eneko Agirre, Mikel Artetxe
Continued pretraining (CPT) is a popular approach to adapt existing large language models (LLMs) to new languages. When doing so, it is common practice to include a portion of Engl…
WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging
Ahmed Elhady, Eneko Agirre, Mikel Artetxe
We introduce WiCkeD, a simple method to increase the complexity of existing multiple-choice benchmarks by randomly replacing a choice with "None of the above", a method often used…
BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation
The Omnilingual MT Team, Pierre Andrews, Mikel Artetxe +14
BOUQuET is a multi-way, multicentric and multi-register/domain dataset and benchmark, and a broader collaborative initiative. This dataset is handcrafted in 8 non-English languages…
Linguini: A benchmark for language-agnostic linguistic reasoning
Eduardo Sánchez, Belen Alastruey, Christophe Ropers +3
We propose a new benchmark to measure a language model's linguistic reasoning skills without relying on pre-existing language-specific knowledge. The test covers 894 questions grou…
BertaQA: How Much Do Language Models Know About Local Culture?
Julen Etxaniz, Gorka Azkune, Aitor Soroa +2
Large Language Models (LLMs) exhibit extensive knowledge about the world, but most evaluations have been limited to global or anglocentric subjects. This raises the question of how…
Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models
Piotr Padlewski, Max Bain, Matthew Henderson +19
We introduce Vibe-Eval: a new open benchmark and framework for evaluating multimodal chat models. Vibe-Eval consists of 269 visual understanding prompts, including 100 of hard diff…
Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models
Reka Team, Aitor Ormazabal, Che Zheng +23
We introduce Reka Core, Flash, and Edge, a series of powerful multimodal language models trained from scratch by Reka. Reka models are able to process and reason with text, images,…
Latxa: An Open Language Model and Evaluation Suite for Basque
Julen Etxaniz, Oscar Sainz, Naiara Perez +6
We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque c…
Towards Artwork Explanation in Large-scale Vision Language Models
Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito +2
Large-scale Vision-Language Models (LVLMs) output text from images and instructions, demonstrating capabilities in text generation and comprehension. However, it has not been clari…
The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak +5
The recent advancements in Large Language Models (LLMs) have garnered widespread acclaim for their remarkable emerging capabilities. However, the issue of hallucination has paralle…
Gender-specific Machine Translation with Large Language Models
Eduardo Sánchez, Pierre Andrews, Pontus Stenetorp +2
While machine translation (MT) systems have seen significant improvements, it is still common for translations to reflect societal biases, such as gender bias. Decoder-only Large L…
Evaluation of Faithfulness Using the Longest Supported Subsequence
Anirudh Mittal, Timo Schick, Mikel Artetxe +1
As increasingly sophisticated language models emerge, their trustworthiness becomes a pivotal issue, especially in tasks such as summarization and question-answering. Ensuring thei…
Do Multilingual Language Models Think Better in English?
Julen Etxaniz, Gorka Azkune, Aitor Soroa +2
Translate-test is a popular technique to improve the performance of multilingual language models. This approach works by translating the input into English using an external machin…
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
Lucas Bandarkar, Davis Liang, Benjamin Muller +7
We present Belebele, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language…
CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models
Aitor Ormazabal, Mikel Artetxe, Eneko Agirre
Methods for adapting language models (LMs) to new tasks and domains have traditionally assumed white-box access to the model, and work by modifying its parameters. However, this is…
Revisiting Machine Translation for Cross-lingual Classification
Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale +2
Machine Translation (MT) has been widely used for cross-lingual classification, either by translating the test set into English and running inference with a monolingual model (tran…
On the Role of Parallel Data in Cross-lingual Transfer Learning
Machel Reid, Mikel Artetxe
While prior work has established that the use of parallel data is conducive for cross-lingual learning, it is unclear if the improvements come from the data itself, or if it is the…
Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training
Kelly Marchisio, Patrick Lewis, Yihong Chen +1
Prior work shows that it is possible to expand pretrained Masked Language Models (MLMs) to new languages by learning a new set of embeddings, while keeping the transformer body fro…
Training Trajectories of Language Models Across Scales
Mengzhou Xia, Mikel Artetxe, Chunting Zhou +5
Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. How do language models…
Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models
Mozes van de Kar, Mengzhou Xia, Danqi Chen +1
Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling. However, this approach is highly sensit…
State-of-the-art generalisation research in NLP: A taxonomy and review
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers +17
The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not…