4 citations · 8 across the 34 of their papers we have counts for
23 papers · 1 filter
A Mechanistic Understanding of Pronoun Fidelity in LLMs
Katharina Trinley, Jesujoba O. Alabi, Dietrich Klakow +1
Faithful and robust pronoun use is important for fair and coherent generations, yet large language models largely fail when multiple referents use different pronouns. To study the…
AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages
Israel Abebe Azime, Jesujoba Oluwadara Alabi, Crystina Zhang +16
Assessing the veracity of a claim made online is a complex and important task with real-world implications. When these claims are directed at communities with limited access to inf…
AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic
Israel Abebe Azime, Abenezer Kebede Angamo, Hana Mekonen Tamiru +4
With the growing emphasis on multilingual and cultural evaluation benchmarks for large language models, language and culture are often treated as synonymous, and performance is com…
Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages
Israel Abebe Azime, Tadesse Destaw Belay, Dietrich Klakow +2
Large language models (LLMs) have demonstrated significant capabilities in solving mathematical problems expressed in natural language. However, multilingual and culturally-grounde…
Colombian Waitresses y Jueces canadienses: Gender and Country Biases in Occupation Recommendations from LLMs
Elisa Forcada Rodríguez, Olatz Perez-de-Viñaspre, Jon Ander Campos +2
One of the goals of fairness research in NLP is to measure and mitigate stereotypical biases that are propagated by NLP systems. However, such work tends to focus on single axes of…
Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
Zena Al-Khalili, Nick Howell, Dietrich Klakow
Assisting LLMs with code generation improved their performance on mathematical reasoning tasks. However, the evaluation of code-assisted LLMs is generally restricted to execution c…
Agree to Disagree? A Meta-Evaluation of LLM Misgendering
Arjun Subramonian, Vagrant Gautam, Preethi Seshadri +3
Numerous methods have been proposed to measure LLM misgendering, including probability-based evaluations (e.g., automatically with templatic sentences) and generation-based evaluat…
Aligned Probing: Relating Toxic Behavior and Model Internals
Andreas Waldis, Vagrant Gautam, Anne Lauscher +2
We introduce aligned probing, a novel interpretability framework that aligns the behavior of language models (LMs), based on their outputs, and their internal representations (inte…
INJONGO: A Multicultural Intent Detection and Slot-filling Dataset for 16 African Languages
Hao Yu, Jesujoba O. Alabi, Andiswa Bukula +19
Slot-filling and intent detection are well-established tasks in Conversational AI. However, current large-scale benchmarks for these tasks often exclude evaluations of low-resource…
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
Xinghao Chen, Zhijing Sun, Wenjin Guo +8
Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought (CoT) prompting. However, CoT prompting greatly increases computational demands, which has prompted g…
Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding
Tadesse Destaw Belay, Israel Abebe Azime, Abinew Ali Ayele +5
Large Language Models (LLMs) show promising learning and reasoning abilities. Compared to other NLP tasks, multilingual and multi-label emotion evaluation tasks are under-explored…
AFRIDOC-MT: Document-level MT Corpus for African Languages
Jesujoba O. Alabi, Israel Abebe Azime, Miaoran Zhang +13
This paper introduces AFRIDOC-MT, a document-level multi-parallel translation dataset covering English and five African languages: Amharic, Hausa, Swahili, Yorùbá, and Zulu. The da…
Utilizing Multimodal Data for Edge Case Robust Call-sign Recognition and Understanding
Alexander Blatt, Dietrich Klakow
Operational machine-learning based assistant systems must be robust in a wide range of scenarios. This hold especially true for the air-traffic control (ATC) domain. The robustness…
WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case
Vagrant Gautam, Julius Steuer, Eileen Bingert +3
While measuring bias and robustness in coreference resolution are important goals, such measurements are only as good as the tools we use to measure them. Winogender Schemas (Rudin…
Improving Semantic Understanding in Speech Language Models via Brain-tuning
Omer Moussa, Dietrich Klakow, Mariya Toneva
Speech language models align with human brain responses to natural language to an impressive degree. However, current models rely heavily on low-level speech features, indicating t…
Human Speech Perception in Noise: Can Large Language Models Paraphrase to Improve It?
Anupama Chingacham, Miaoran Zhang, Vera Demberg +1
Large Language Models (LLMs) can generate text by transferring style attributes like formality resulting in formal or informal text. However, instructing LLMs to generate text that…
Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning
Pin-Jie Lin, Miaoran Zhang, Marius Mosbach +1
Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. In this work, we experiment with 130 source-target task comb…
Joint vs Sequential Speaker-Role Detection and Automatic Speech Recognition for Air-traffic Control
Alexander Blatt, Aravind Krishnan, Dietrich Klakow
Utilizing air-traffic control (ATC) data for downstream natural-language processing tasks requires preprocessing steps. Key steps are the transcription of the data via automatic sp…
On the Encoding of Gender in Transformer-based ASR Representations
Aravind Krishnan, Badr M. Abdullah, Dietrich Klakow
While existing literature relies on performance differences to uncover gender biases in ASR models, a deeper analysis is essential to understand how gender is encoded and utilized…
From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP
Marius Mosbach, Vagrant Gautam, Tomás Vergara-Browne +2
Interpretability and analysis (IA) research is a growing subfield within NLP with the goal of developing a deeper understanding of the behavior or inner workings of NLP systems and…
Understanding "Democratization" in NLP and ML Research
Arjun Subramonian, Vagrant Gautam, Dietrich Klakow +1
Recent improvements in natural language processing (NLP) and machine learning (ML) and increased mainstream adoption have led to researchers frequently discussing the "democratizat…
Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?
Dawei Zhu, Pinzhen Chen, Miaoran Zhang +3
Traditionally, success in multilingual machine translation can be attributed to three key factors in training data: large volume, diverse translation directions, and high quality.…
A Preference-driven Paradigm for Enhanced Translation with Large Language Models
Dawei Zhu, Sony Trenous, Xiaoyu Shen +3
Recent research has shown that large language models (LLMs) can achieve remarkable translation performance through supervised fine-tuning (SFT) using only a small amount of paralle…