1 citations · 1 across the 10 of their papers we have counts for
13 papers
Calibrating LLM Judges for Human and AI Conversations
Maike Züfle, Patrícia Schmidtová, Vilém Zouhar +9
Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of convers…
Controlling Backchannels in Streamable Full-duplex Models
Maike Züfle, Peter Polák, Sefik Emre Eskimez +3
Backchannels, brief acknowledgements like "uh-huh" produced while the other party may still be talking, are central to natural conversation, but full-duplex spoken dialogue models…
Distributional Metrics for Evaluating Spoken Conversational Systems
Shree Harsha Bokkahalli Satish, Erica Cooper, Patrícia Schmidtová +4
Evaluating conversational systems is a difficult and unresolved problem. We introduce the Conversational Distribution Score (CDS), which compares distributions of conversational be…
An Evaluation Framework for Text-to-Speech Voice Reconstruction
Ariadna Sanchez, Christoph Minixhofer, Korin Richmond +3
Voice reconstruction using Text-to-Speech (TTS) offers a communication method for people with speech disorders, which aims to retain their speaker identity while improving intellig…
Automatic Labelling of Speech Translation Errors
Dominik Macháček, Maike Züfle, Ondrej Klejch
Errors in speech translations reduce trustworthiness of Speech Translation (ST) systems and can have serious consequences. Yet currently there is no established methodology for eva…
From Seeing it to Experiencing it: Interactive Evaluation of Intersectional Voice Bias in Human-AI Speech Interaction
Shree Harsha Bokkahalli Satish, Maria Teleki, Christoph Minixhofer +3
SpeechLLMs process spoken language directly from audio, but accent and vocal identity cues can lead to biased behaviour. Current bias evaluations often miss how such bias manifests…
The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs
Shree Harsha Bokkahalli Satish, Christoph Minixhofer, Maria Teleki +5
Speech Large Language Models (SpeechLLMs) process spoken input directly, retaining cues such as accent and perceived gender that were previously removed in cascaded pipelines. This…
F-Actor: Controllable Conversational Behaviour in Full-Duplex Models
Maike Züfle, Ondrej Klejch, Nicholas Sanders +3
Spoken conversational systems require more than accurate speech generation to have human-like conversations: to feel natural and engaging, they must produce conversational behaviou…
TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems
Christoph Minixhofer, Ondrej Klejch, Peter Bell
Evaluation of Text to Speech (TTS) systems is challenging and resource-intensive. Subjective metrics such as Mean Opinion Score (MOS) are not easily comparable between works. Objec…
Speech Recognition for Automatically Assessing Afrikaans and isiXhosa Preschool Oral Narratives
Christiaan Jacobs, Annelien Smith, Daleen Klop +3
We develop automatic speech recognition (ASR) systems for stories told by Afrikaans and isiXhosa preschool children. Oral narratives provide a way to assess children's language dev…
TTSDS -- Text-to-Speech Distribution Score
Christoph Minixhofer, Ondřej Klejch, Peter Bell
Many recently published Text-to-Speech (TTS) systems produce audio close to real speech. However, TTS evaluation needs to be revisited to make sense of the results obtained with th…
Evaluating and reducing the distance between synthetic and real speech distributions
Christoph Minixhofer, Ondřej Klejch, Peter Bell
While modern Text-to-Speech (TTS) systems can produce natural-sounding speech, they remain unable to reproduce the full diversity found in natural speech data. We consider the dist…
Mask-combine Decoding and Classification Approach for Punctuation Prediction with real-time Inference Constraints
Christoph Minixhofer, Ondřej Klejch, Peter Bell
In this work, we unify several existing decoding strategies for punctuation prediction in one framework and introduce a novel strategy which utilises multiple predictions at each w…