14 papers
Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training
JiWoo Hwang, Lukas Edman, Alexander Fraser
We investigate two aspects of multilingual pre-training with limited data, namely how language exposure is distributed across English, Dutch, and Chinese, and how that exposure is…
Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages
Irem Ecem Yelkanat, Faeze Ghorbanpour, Alexander Fraser
Annotator disagreement is increasingly recognized as a meaningful property of subjective tasks, rather than merely noise to be removed. Predicting which items are likely to be cont…
Retrieval Disagreement as a Signal: Diagnosing and Steering Multilingual Dense Embedders
Faeze Ghorbanpour, Shaghayegh Kolli, Jana Diesner +1
Multilingual dense retrievers place all languages in one shared space so that a question finds relevant documents, regardless of which language they are written in. We show that th…
Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops
Faeze Ghorbanpour, Constanza Fierro, Alexander Fraser +1
Large Language Models (LLMs) often answer the same factual question differently across languages. We study whether cross-lingual latent-space intervention can reduce this inconsist…
Toppling the Hierarchy in Byte-level Language Modeling
Lukas Edman, Alexander Fraser
This work examines recent byte-level models and their failure to perfectly manipulate characters. State-of-the-art byte-level models use a hierarchical structure, starting at the b…
From Specialization to Generalization: Instruction-tuned LLMs for Robust Harmful Content Mitigation
Lukas Edman, Daryna Dementieva, Alexander Fraser
Large language models (LLMs) demonstrate impressive performance across a wide range of general NLP tasks; however, their effectiveness in sensitive domains, such as hate speech det…
Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations
Mohammad Amine Jradi, Faeze Ghorbanpour, Alexander Fraser
Hate speech annotation is costly, subjective, and prone to annotator disagreement, making large-scale dataset construction challenging. We systematically analyze how well large lan…
PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions
Ruhallah Niazi, Faeze Ghorbanpour, Alexander Fraser
Despite impressive multilingual capabilities, large language models (LLMs) remain poorly evaluated on literary knowledge in non-English languages. We introduce PersLitEval, a bench…
On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputs
Faeze Ghorbanpour, Alexander Fraser
Large language models (LLMs) increasingly operate on long inputs, yet their behavior when harmful sentences are sparsely embedded within such inputs remains poorly understood. We p…
Data-Efficient Hate Speech Detection via Cross-Lingual Nearest Neighbor Retrieval with Limited Labeled Data
Faeze Ghorbanpour, Daryna Dementieva, Alexander Fraser
Considering the importance of detecting hateful language, labeled hate speech data is expensive and time-consuming to collect, particularly for low-resource languages. Prior work h…
Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study
Faeze Ghorbanpour, Daryna Dementieva, Alexander Fraser
Despite growing interest in automated hate speech detection, most existing approaches overlook the linguistic diversity of online content. Multilingual instruction-tuned large lang…
EXECUTE: A Multilingual Benchmark for LLM Token Understanding
Lukas Edman, Helmut Schmid, Alexander Fraser
The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE.…
Are BabyLMs Second Language Learners?
Lukas Edman, Lisa Bylinina, Faeze Ghorbanpour +1
This paper describes a linguistically-motivated approach to the 2024 edition of the BabyLM Challenge (Warstadt et al. 2023). Rather than pursuing a first language learning (L1) par…
CUTE: Measuring LLMs' Understanding of Their Tokens
Lukas Edman, Helmut Schmid, Alexander Fraser
Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direc…