15 citations · 49 across the 45 of their papers we have counts for
22 papers · 1 filter
Voice Memory for Agentic Speech Recognition
Chao-Han Huck Yang, Zih-Ching Chen, Piotr Zelasko +3
We present Voice Memory, a inference-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per-domain memory.md and decides per utterance wh…
Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs
Nithin Rao Koluguri, Sasha Meister, Nikolay Karpov +4
Popular ASR test sets adopt inconsistent conventions for numbers, disfluencies, entities, and casing, while standard normalizers erase the format distinctions users care about. Cur…
Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech
Siqi Ouyang, Shuoyang Ding, Oleksii Hrinchuk +4
Simultaneous speech translation (SST) generates translations while receiving partial speech input. Recent advances show that large language models (LLMs) can substantially improve…
Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST
Monica Sekoyan, Nithin Rao Koluguri, Nune Tadevosyan +5
This report introduces Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST). Built with a FastConformer encod…
Open Automatic Speech Recognition Models for Classical and Modern Standard Arabic
Lilit Grigoryan, Nikolay Karpov, Enas Albasiri +2
Despite Arabic being one of the most widely spoken languages, the development of Arabic Automatic Speech Recognition (ASR) systems faces significant challenges due to the language'…
Word Level Timestamp Generation for Automatic Speech Recognition and Translation
Ke Hu, Krishna Puvvada, Elena Rastorgueva +7
We introduce a data-driven approach for enabling word-level timestamp prediction in the Canary model. Accurate timestamp information is crucial for a variety of downstream tasks su…
Granary: Speech Recognition and Translation Dataset in 25 European Languages
Nithin Rao Koluguri, Monica Sekoyan, George Zelenfroynd +12
Multi-task and multilingual approaches benefit large models, yet speech processing for low-resource languages remains underexplored due to data scarcity. To address this, we presen…
SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model
Ke Hu, Ehsan Hosseini-Asl, Chen Chen +7
Spoken dialogue is an intuitive form of human-computer interaction, yet current speech language models often remain constrained to turn-based exchanges, lacking real-time adaptabil…
Training and Inference Efficiency of Encoder-Decoder Speech Models
Piotr Żelasko, Kunal Dhawan, Daniel Galvez +7
Attention encoder-decoder model architecture is the backbone of several recent top performing foundation speech models: Whisper, Seamless, OWSM, and Canary-1B. However, the reporte…
NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Model
Yen-Ting Lin, Zhehuai Chen, Piotr Zelasko +11
Construction of a general-purpose post-recognition error corrector poses a crucial question: how can we most effectively train a model on a large mixture of domain datasets? The an…
Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition
Chao-Han Huck Yang, Taejin Park, Yuan Gong +18
Given recent advances in generative AI technology, a key question is how large language models (LLMs) can enhance acoustic modeling tasks using text decoding results from a frozen,…
Anticipating Future with Large Language Model for Simultaneous Machine Translation
Siqi Ouyang, Oleksii Hrinchuk, Zhehuai Chen +4
Simultaneous machine translation (SMT) takes streaming input utterances and incrementally produces target text. Existing SMT methods mainly use the partial utterance that has alrea…
VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning
Yifan Peng, Krishna C. Puvvada, Zhehuai Chen +7
Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models (SpeechLMs). Earlier SpeechLMs focused on…
EMMeTT: Efficient Multimodal Machine Translation Training
Piotr Żelasko, Zhehuai Chen, Mengru Wang +7
A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses…
Chain-of-Thought Prompting for Speech Translation
Ke Hu, Zhehuai Chen, Chao-Han Huck Yang +5
Large language models (LLMs) have demonstrated remarkable advancements in language understanding and generation. Building on the success of text-based LLMs, recent research has ada…
Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Models
Somshubra Majumdar, Vahid Noroozi, Mehrzad Samadi +6
Large Language Models (LLMs) require high quality instruction data for effective alignment, particularly in code generation tasks where expert curated datasets are expensive to pro…
Romanization Encoding For Multilingual ASR
Wen Ding, Fei Jia, Hainan Xu +3
We introduce romanization encoding for script-heavy languages to optimize multilingual and code-switching Automatic Speech Recognition (ASR) systems. By adopting romanization encod…
BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5
Zhehuai Chen, He Huang, Oleksii Hrinchuk +5
Incorporating speech understanding capabilities into pretrained large-language models has become a vital research direction (SpeechLLM). The previous architectures can be categoriz…
Transducers with Pronunciation-aware Embeddings for Automatic Speech Recognition
Hainan Xu, Zhehuai Chen, Fei Jia +1
This paper proposes Transducers with Pronunciation-aware Embeddings (PET). Unlike conventional Transducers where the decoder embeddings for different tokens are trained independent…
RULER: What's the Real Context Size of Your Long-Context Language Models?
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman +5
The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adop…
Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition
Vahid Noroozi, Somshubra Majumdar, Ankur Kumar +2
In this paper, we propose an efficient and accurate streaming speech recognition model based on the FastConformer architecture. We adapted the FastConformer architecture for stream…
SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation
Zhehuai Chen, He Huang, Andrei Andrusenko +6
We present a novel Speech Augmented Language Model (SALM) with {\em multitask} and {\em in-context} learning capabilities. SALM comprises a frozen text LLM, a audio encoder, a moda…