215 citations · 229 across the 19 of their papers we have counts for
24 papers
CRITICS - Critical Science Without Borders: Language Models to Promote Critical Thinking in Science Education
Rodrigo Agerri, Itziar Aldabe, Elena Cabrio +10
The CRITICS project addresses science accessibility and literacy by converging advanced Machine Translation (MT) based on Large Language Models (LLMs) with educational technology.…
Your Voice Cloning System is Secretly a Voice Anonymizer
Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak +1
Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice clo…
The RAT: A Unified Bayesian Model for RAG Evaluation
Pius von Däniken, Felix Matthias Saaro, Mark Cieliebak +1
Evaluating Retrieval-Augmented Generation (RAG) systems requires assessing not only end-to-end correctness but also how individual components interact and how errors propagate thro…
SwissGPC v1.0 -- The Swiss German Podcasts Corpus
Samuel Stucki, Mark Cieliebak, Jan Deriu
We present SwissGPC v1.0, the first mid-to-large-scale corpus of spontaneous Swiss German speech, developed to support research in ASR, TTS, dialect identification, and related fie…
Voice Adaptation for Swiss German
Samuel Stucki, Jan Deriu, Mark Cieliebak
This work investigates the performance of Voice Adaptation models for Swiss German dialects, i.e., translating Standard German text to Swiss German dialect speech. For this, we pre…
ViClaim: A Multilingual Multilabel Dataset for Automatic Claim Detection in Videos
Patrick Giedemann, Pius von Däniken, Jan Deriu +3
The growing influence of video content as a medium for communication and misinformation underscores the urgent need for effective tools to analyze claims in multilingual and multi-…
A Measure of the System Dependence of Automated Metrics
Pius von Däniken, Jan Deriu, Mark Cieliebak
Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations. These metrics are typically ass…
Error-preserving Automatic Speech Recognition of Young English Learners' Language
Janick Michot, Manuela Hürlimann, Jan Deriu +3
One of the central skills that language learners need to practice is speaking the language. Currently, students in school do not get enough speaking opportunities and lack conversa…
Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluation
Pius von Däniken, Jan Deriu, Don Tuggener +1
Generative AI systems have become ubiquitous for all kinds of modalities, which makes the issue of the evaluation of such models more pressing. One popular approach is preference r…
Dialect Transfer for Swiss German Speech Translation
Claudio Paonessa, Yanick Schraner, Jan Deriu +3
This paper investigates the challenges in building Swiss German speech translation systems, specifically focusing on the impact of dialect diversity and differences between Swiss G…
Correction of Errors in Preference Ratings from Automated Metrics for Text Generation
Jan Deriu, Pius von Däniken, Don Tuggener +1
A major challenge in the field of Text Generation is evaluation: Human evaluations are cost-intensive, and automated metrics often display considerable disagreement with human judg…
STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions
Michel Plüss, Jan Deriu, Yanick Schraner +8
We present STT4SG-350 (Speech-to-Text for Swiss German), a corpus of Swiss German speech, annotated with Standard German text at the sentence level. The data is collected using a w…
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
Anya Belz, Craig Thomson, Ehud Reiter +39
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/le…
On the Effectiveness of Automated Metrics for Text Generation Systems
Pius von Däniken, Jan Deriu, Don Tuggener +1
A major challenge in the field of Text Generation is evaluation because we lack a sound theory that can be leveraged to extract guidelines for evaluation campaigns. In this work, w…
SDS-200: A Swiss German Speech to Standard German Text Corpus
Michel Plüss, Manuela Hürlimann, Marc Cuny +10
We present SDS-200, a corpus of Swiss German dialectal speech with Standard German text translations, annotated with dialect, age, and gender information of the speakers. The datas…
Probing the Robustness of Trained Metrics for Conversational Dialogue Systems
Jan Deriu, Don Tuggener, Pius von Däniken +1
This paper introduces an adversarial method to stress-test trained metrics to evaluate conversational dialogue systems. The method leverages Reinforcement Learning to find response…
Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems
Jan Deriu, Don Tuggener, Pius von Däniken +6
The lack of time-efficient and reliable evaluation methods hamper the development of conversational dialogue systems (chatbots). Evaluations requiring humans to converse with chatb…
DoQA -- Accessing Domain-Specific FAQs via Conversational QA
Jon Ander Campos, Arantxa Otegi, Aitor Soroa +3
The goal of this work is to build conversational Question Answering (QA) interfaces for the large body of domain-specific information available in FAQ sites. We present DoQA, a dat…
A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation
Jan Deriu, Katsiaryna Mlynchyk, Philippe Schläpfer +6
In this paper, we introduce a novel methodology to efficiently construct a corpus for question answering over structured data. For this, we introduce an intermediate representation…
Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvement
Jan Deriu, Mark Cieliebak
We present "AutoJudge", an automated evaluation method for conversational dialogue systems. The method works by first generating dialogues based on self-talk, i.e. dialogue systems…
Correlating Twitter Language with Community-Level Health Outcomes
Arno Schneuwly, Ralf Grubenmann, Séverine Rion Logean +2
We study how language on social media is linked to diseases such as atherosclerotic heart disease (AHD), diabetes and various types of cancer. Our proposed model leverages state-of…
Towards Integration of Statistical Hypothesis Tests into Deep Neural Networks
Ahmad Aghaebrahimian, Mark Cieliebak
We report our ongoing work about a new deep architecture working in tandem with a statistical test procedure for jointly training texts and their label descriptions for multi-label…
Survey on Evaluation Methods for Dialogue Systems
Jan Deriu, Alvaro Rodrigo, Arantxa Otegi +4
In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue syst…
Leveraging Large Amounts of Weakly Supervised Data for Multi-Language Sentiment Classification
Jan Deriu, Aurelien Lucchi, Valeria De Luca +5
This paper presents a novel approach for multi-lingual sentiment classification in short texts. This is a challenging task as the amount of training data in languages other than En…