20 papers
Calibrating LLM Judges for Human and AI Conversations
Maike Züfle, Patrícia Schmidtová, Vilém Zouhar +9
Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of convers…
Controlling Backchannels in Streamable Full-duplex Models
Maike Züfle, Peter Polák, Sefik Emre Eskimez +3
Backchannels, brief acknowledgements like "uh-huh" produced while the other party may still be talking, are central to natural conversation, but full-duplex spoken dialogue models…
Speech-to-SOAP: End-to-End Summarization of Medical Dialogues: KIT@BeTraC 2026
Enes Yavuz Ugan, Fabian Retkowski, Yuka Ko +4
With the advent of Large Language Models and its instruction following capabilities a promising application is the task of summarization. Within this domain of task the extractive…
The Role of Disfluencies in Speech Translation
Maike Züfle, Maria Teleki, Fabian Retkowski +5
Current speech translation systems, including SpeechLLMs, are trained on cleaned text and tend to strip disfluencies like filled pauses and false starts rather than translate them.…
Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026
Enes Yavuz Ugan, Maike Züfle, Yuka Ko +5
With the advent of Large Language Models, single-task and token-based multi-task models have evolved into instruction-based systems that infer task and target language implicitly f…
When Helpful Context Leaks: Privacy Risks in Domain-Adapted ASR
Maike Züfle, Jan Niehues
SpeechLLMs are increasingly deployed in professional settings where domain customisation is standard practice: users supply context in prompts with sensitive information, fine-tune…
Why We Need Speech to Evaluate Speech Translation
Maike Züfle, Danni Liu, Vilém Zouhar +1
Speech translation models are increasingly capable of preserving speech-specific information (e.g., speaker gender, prosody, and emphasis), yet evaluation metrics remain blind to s…
Do What I Say: A Spoken Prompt Dataset for Instruction-Following
Maike Züfle, Sara Papi, Fabian Retkowski +5
Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompts, which may not reflect real-wo…
Beyond Transcripts: A Renewed Perspective on Audio Chaptering
Fabian Retkowski, Maike Züfle, Thai Binh Nguyen +2
Audio chaptering, the task of segmenting long-form audio into coherent sections, is increasingly important for navigating podcasts, lectures, and videos. Despite its relevance, res…
F-Actor: Controllable Conversational Behaviour in Full-Duplex Models
Maike Züfle, Ondrej Klejch, Nicholas Sanders +3
Spoken conversational systems require more than accurate speech generation to have human-like conversations: to feel natural and engaging, they must produce conversational behaviou…
Talk2Ref: A Dataset for Reference Prediction from Scientific Talks
Frederik Broy, Maike Züfle, Jan Niehues
Scientific talks are a growing medium for disseminating research, and automatically identifying relevant literature that grounds or enriches a talk would be highly valuable for res…
Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks
Supriti Sinhamahapatra, Jan Niehues
State-of-the-art (SOTA) Automatic Speech Recognition (ASR) systems primarily rely on acoustic information while disregarding additional multi-modal context. However, visual informa…
MuSaG: A Multimodal German Sarcasm Dataset with Full-Modal Annotations
Aaron Scott, Maike Züfle, Jan Niehues
Sarcasm is a complex form of figurative language in which the intended meaning contradicts the literal one. Its prevalence in social media and popular culture poses persistent chal…
COMET-poly: Machine Translation Metric Grounded in Other Candidates
Maike Züfle, Vilém Zouhar, Tu Anh Dinh +3
Automated metrics for machine translation attempt to replicate human judgment. Unlike humans, who often assess a translation in the context of multiple alternatives, these metrics…
MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks
Sara Papi, Maike Züfle, Marco Gaido +5
Recent advances in large language models have laid the foundation for multimodal LLMs (MLLMs), which unify text, speech, and vision within a single framework. As these models are r…
KIT's Offline Speech Translation and Instruction Following Submission for IWSLT 2025
Sai Koneru, Maike Züfle, Thai-Binh Nguyen +3
The scope of the International Workshop on Spoken Language Translation (IWSLT) has recently broadened beyond traditional Speech Translation (ST) to encompass a wider array of tasks…
Summarizing Speech: A Comprehensive Survey
Fabian Retkowski, Maike Züfle, Andreas Sudmann +4
Speech summarization has become an essential tool for efficiently managing and accessing the growing volume of spoken and audiovisual content. However, despite its increasing impor…
NUTSHELL: A Dataset for Abstract Generation from Scientific Talks
Maike Züfle, Sara Papi, Beatrice Savoldi +3
Scientific communication is receiving increasing attention in natural language processing, especially to help researches access, summarize, and generate content. One emerging appli…
Early-Exit and Instant Confidence Translation Quality Estimation
Vilém Zouhar, Maike Züfle, Beni Egressy +3
Quality estimation is omnipresent in machine translation, for both evaluation and generation. Unfortunately, quality estimation models are often opaque and computationally expensiv…
Contrastive Learning for Task-Independent SpeechLLM-Pretraining
Maike Züfle, Jan Niehues
Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward. Direct task-specific fine-t…