7 papers
Factorized Delayed Streams Modeling for LLM-based Streaming ASR
Tatsunari Takagi, Kai Washizaki, Atsushi Kojima +3
Delayed Streams Modeling (DSM) enables LLM-based streaming automatic speech recognition (ASR) by aligning acoustic and text streams on a common timeline. DSM adds the padding token…
Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models
Tomoya Mizumoto, Yusuke Fujita, Hao Shi +3
Dialogue systems based on large language models (LLMs) have advanced significantly in recent years. However, dialectal variation remains a major challenge, particularly for systems…
Serialized Output Prompting for Large Language Model-based Multi-Talker Speech Recognition
Hao Shi, Yusuke Fujita, Tomoya Mizumoto +3
Prompts are crucial for task definition and for improving the performance of large language models (LLM)-based systems. However, existing LLM-based multi-talker (MT) automatic spee…
AC/DC: LLM-based Audio Comprehension via Dialogue Continuation
Yusuke Fujita, Tomoya Mizumoto, Atsushi Kojima +2
We propose an instruction-following audio comprehension model that leverages the dialogue continuation ability of large language models (LLMs). Instead of directly generating targe…
OWSM-Biasing: Contextualizing Open Whisper-Style Speech Models for Automatic Speech Recognition with Dynamic Vocabulary
Yui Sudo, Yusuke Fujita, Atsushi Kojima +2
Speech foundation models (SFMs), such as Open Whisper-Style Speech Models (OWSM), are trained on massive datasets to achieve accurate automatic speech recognition. However, even SF…
Self-Supervised Learning for Multi-Channel Neural Transducer
Atsushi Kojima
Self-supervised learning, such as with the wav2vec 2.0 framework significantly improves the accuracy of end-to-end automatic speech recognition (ASR). Wav2vec 2.0 has been applied…
Intermediate direct preference optimization
Atsushi Kojima
We propose the intermediate direct preference optimization (DPO) method to calculate the DPO loss at selected intermediate layers as an auxiliary loss for finetuning large language…