15 papers
From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios
Thai-Binh Nguyen, Zhaolin Li, Jan Niehues +1
Humans have the remarkable ability to engage in spontaneous informal conversations and selectively attend to individual speakers while filtering out competing speech from nearby co…
MUSCAT: MUltilingual, SCientific ConversATion Benchmark
Supriti Sinhamahapatra, Thai-Binh Nguyen, YiÄit OÄuz +3
The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages, creating the experience as though everyone were…
Multimodal In-context Learning for ASR of Low-resource Languages
Zhaolin Li, Jan Niehues
Automatic speech recognition (ASR) still covers only a small fraction of the world's languages, mainly due to supervised data scarcity. In-context learning (ICL) with large languag…
OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion
Sai Koneru, Matthias Huck, Jan Niehues
There has been significant progress in open-source text-only translation large language models (LLMs) with better language coverage and quality. However, these models can be only u…
Sigmoid Head for Quality Estimation under Language Ambiguity
Tu Anh Dinh, Jan Niehues
Language model (LM) probability is not a reliable quality estimator, as natural language is ambiguous. When multiple output options are valid, the model's probability distribution…
From Slides to Chatbots: Enhancing Large Language Models with University Course Materials
Tu Anh Dinh, Philipp Nicolas Schumacher, Jan Niehues
Large Language Models (LLMs) have advanced rapidly in recent years. One application of LLMs is to support student learning in educational settings. However, prior work has shown th…