4 citations · 8 across the 34 of their papers we have counts for
34 papers
Fork-Think with Confidence
Zena Al-Khalili, Rafi Hakim, Dietrich Klakow +1
Parallel thinking has enjoyed great success for boosting LLM performance on reasoning tasks without the need for any re-training. However, existing methods follow a think-first-the…
A Mechanistic Understanding of Pronoun Fidelity in LLMs
Katharina Trinley, Jesujoba O. Alabi, Dietrich Klakow +1
Faithful and robust pronoun use is important for fair and coherent generations, yet large language models largely fail when multiple referents use different pronouns. To study the…
AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages
Israel Abebe Azime, Jesujoba Oluwadara Alabi, Crystina Zhang +16
Assessing the veracity of a claim made online is a complex and important task with real-world implications. When these claims are directed at communities with limited access to inf…
On Optimizing Multimodal Jailbreaks for Spoken Language Models
Aravind Krishnan, Karolina Stańczak, Dietrich Klakow
As Spoken Language Models (SLMs) integrate speech and text modalities, they inherit the safety vulnerabilities of their LLM backbone while introducing an expanded attack surface. S…
Split Personality Training: Revealing Latent Knowledge Through Alternate Personalities
Florian Dietz, William Wale, Oscar Gilg +5
Detecting misalignment in large language models is challenging because models may learn to conceal misbehavior during training. Standard auditing techniques fall short: black-box m…
AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic
Israel Abebe Azime, Abenezer Kebede Angamo, Hana Mekonen Tamiru +4
With the growing emphasis on multilingual and cultural evaluation benchmarks for large language models, language and culture are often treated as synonymous, and performance is com…
PricingLogic: Evaluating LLMs Reasoning on Complex Tourism Pricing Tasks
Yunuo Liu, Dawei Zhu, Zena Al-Khalili +5
We present PricingLogic, the first benchmark that probes whether Large Language Models(LLMs) can reliably automate tourism-related prices when multiple, overlapping fare rules appl…
Decoding Partial Differential Equations: Cross-Modal Adaptation of Decoder-only Models to PDEs
Paloma García-de-Herreros, Philipp Slusallek, Dietrich Klakow +1
While large language models are primarily used on natural language tasks, they have also shown great promise when adapted to new modalities, e.g., for scientific machine learning t…
Accept or Deny? Evaluating LLM Fairness and Performance in Loan Approval across Table-to-Text Serialization Approaches
Israel Abebe Azime, Deborah D. Kanubala, Tejumade Afonja +4
Large Language Models (LLMs) are increasingly employed in high-stakes decision-making tasks, such as loan approvals. While their applications expand across domains, LLMs struggle t…
Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages
Israel Abebe Azime, Tadesse Destaw Belay, Dietrich Klakow +2
Large language models (LLMs) have demonstrated significant capabilities in solving mathematical problems expressed in natural language. However, multilingual and culturally-grounde…
Colombian Waitresses y Jueces canadienses: Gender and Country Biases in Occupation Recommendations from LLMs
Elisa Forcada Rodríguez, Olatz Perez-de-Viñaspre, Jon Ander Campos +2
One of the goals of fairness research in NLP is to measure and mitigate stereotypical biases that are propagated by NLP systems. However, such work tends to focus on single axes of…
Implementing Rational Choice Functions with LLMs and Measuring their Alignment with User Preferences
Anna Karnysheva, Christian Drescher, Dietrich Klakow
As large language models (LLMs) become integral to intelligent user interfaces (IUIs), their role as decision-making agents raises critical concerns about alignment. Although exten…
Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
Zena Al-Khalili, Nick Howell, Dietrich Klakow
Assisting LLMs with code generation improved their performance on mathematical reasoning tasks. However, the evaluation of code-assisted LLMs is generally restricted to execution c…
Agree to Disagree? A Meta-Evaluation of LLM Misgendering
Arjun Subramonian, Vagrant Gautam, Preethi Seshadri +3
Numerous methods have been proposed to measure LLM misgendering, including probability-based evaluations (e.g., automatically with templatic sentences) and generation-based evaluat…
Aligned Probing: Relating Toxic Behavior and Model Internals
Andreas Waldis, Vagrant Gautam, Anne Lauscher +2
We introduce aligned probing, a novel interpretability framework that aligns the behavior of language models (LMs), based on their outputs, and their internal representations (inte…
INJONGO: A Multicultural Intent Detection and Slot-filling Dataset for 16 African Languages
Hao Yu, Jesujoba O. Alabi, Andiswa Bukula +19
Slot-filling and intent detection are well-established tasks in Conversational AI. However, current large-scale benchmarks for these tasks often exclude evaluations of low-resource…
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
Xinghao Chen, Zhijing Sun, Wenjin Guo +8
Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought (CoT) prompting. However, CoT prompting greatly increases computational demands, which has prompted g…
Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding
Tadesse Destaw Belay, Israel Abebe Azime, Abinew Ali Ayele +5
Large Language Models (LLMs) show promising learning and reasoning abilities. Compared to other NLP tasks, multilingual and multi-label emotion evaluation tasks are under-explored…
AFRIDOC-MT: Document-level MT Corpus for African Languages
Jesujoba O. Alabi, Israel Abebe Azime, Miaoran Zhang +13
This paper introduces AFRIDOC-MT, a document-level multi-parallel translation dataset covering English and five African languages: Amharic, Hausa, Swahili, Yorùbá, and Zulu. The da…
IGC: Integrating a Gated Calculator into an LLM to Solve Arithmetic Tasks Reliably and Efficiently
Florian Dietz, Dietrich Klakow
Solving arithmetic tasks is a simple and fundamental skill, yet modern Large Language Models (LLMs) have great difficulty with them. We introduce the Integrated Gated Calculator (I…
Utilizing Multimodal Data for Edge Case Robust Call-sign Recognition and Understanding
Alexander Blatt, Dietrich Klakow
Operational machine-learning based assistant systems must be robust in a wide range of scenarios. This hold especially true for the air-traffic control (ATC) domain. The robustness…
WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case
Vagrant Gautam, Julius Steuer, Eileen Bingert +3
While measuring bias and robustness in coreference resolution are important goals, such measurements are only as good as the tools we use to measure them. Winogender Schemas (Rudin…
Improving Semantic Understanding in Speech Language Models via Brain-tuning
Omer Moussa, Dietrich Klakow, Mariya Toneva
Speech language models align with human brain responses to natural language to an impressive degree. However, current models rely heavily on low-level speech features, indicating t…
Human Speech Perception in Noise: Can Large Language Models Paraphrase to Improve It?
Anupama Chingacham, Miaoran Zhang, Vera Demberg +1
Large Language Models (LLMs) can generate text by transferring style attributes like formality resulting in formal or informal text. However, instructing LLMs to generate text that…