17 citations · 48 across the 21 of their papers we have counts for
16 papers · 1 filter
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
Nafiseh Nikeghbal, Amir Hossein Kargaran, Shaghayegh Kolli +1
Standard accuracy benchmarks evaluate whether large language models (LLMs) reach correct answers. However, they do not test whether models maintain that answer when challenged by a…
GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts
Amir Hossein Kargaran, Nafiseh Nikeghbal, Jana Diesner +2
Optical character recognition (OCR) has advanced rapidly with the rise of vision-language models, yet evaluation has remained concentrated on a small cluster of high- and mid-resou…
CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs
Nafiseh Nikeghbal, Amir Hossein Kargaran, Jana Diesner
Improvements in model construction, including fortified safety guardrails, allow Large language models (LLMs) to increasingly pass standard safety checks. However, LLMs sometimes s…
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec +7
Pre-training state-of-the-art large language models (LLMs) requires vast amounts of clean and diverse text data. While the open development of large high-quality English pre-traini…
How Programming Concepts and Neurons Are Shared in Code Language Models
Amir Hossein Kargaran, Yihong Liu, François Yvon +1
Several studies have explored the mechanisms of large language models (LLMs) in coding tasks, but most have focused on programming languages (PLs) in a monolingual setting. In this…
Tracing Multilingual Factual Knowledge Acquisition in Pretraining
Yihong Liu, Mingyang Wang, Amir Hossein Kargaran +5
Large Language Models (LLMs) are capable of recalling multilingual factual knowledge present in their pretraining data. However, most studies evaluate only the final model, leaving…
On Relation-Specific Neurons in Large Language Models
Yihong Liu, Runsheng Chen, Lea Hirlimann +6
In large language models (LLMs), certain \emph{neurons} can store distinct pieces of knowledge learned during pretraining. While factual knowledge typically appears as a combinatio…
GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages
Amir Hossein Kargaran, François Yvon, Hinrich Schütze
The need for large text corpora has increased with the advent of pretrained language models and, in particular, the discovery of scaling laws for these models. Most available corpo…
MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment
Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal +3
English-centric large language models (LLMs) often show strong multilingual capabilities. However, their multilingual performance remains unclear and is under-evaluated for many ot…
How Transliterations Improve Crosslingual Alignment
Yihong Liu, Mingyang Wang, Amir Hossein Kargaran +6
Recent studies have shown that post-aligning multilingual pretrained language models (mPLMs) using alignment objectives on both original and transliterated data can improve crossli…
MaskLID: Code-Switching Language Identification through Iterative Masking
Amir Hossein Kargaran, François Yvon, Hinrich Schütze
We present MaskLID, a simple, yet effective, code-switching (CS) language identification (LID) method. MaskLID does not require any training and is designed to complement current h…
GlotLID: Language Identification for Low-Resource Languages
Amir Hossein Kargaran, Ayyoob Imani, François Yvon +1
Several recent papers have published good solutions for language identification (LID) for about 300 high-resource and medium-resource languages. However, there is no LID available…
GlotScript: A Resource and Tool for Low Resource Writing System Identification
Amir Hossein Kargaran, François Yvon, Hinrich Schütze
We present GlotScript, an open resource and tool for low resource writing system identification. GlotScript-R is a resource that provides the attested writing systems for more than…
Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages
Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran +8
The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., making them better for about 100 languages. We instead scale LLMs horizontally: we cr…
MenuCraft: Interactive Menu System Design with Large Language Models
Amir Hossein Kargaran, Nafiseh Nikeghbal, Abbas Heydarnoori +1
Menu system design for user interfaces is a challenging task involving many design options and various human factors. For example, one crucial factor that designers need to conside…