40 citations · 109 across the 27 of their papers we have counts for
28 papers · 1 filter
Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems
Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung +3
Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval sy…
Multimodal Modeling For Spoken Language Identification
Shikhar Bharadwaj, Min Ma, Shikhar Vashishth +10
Spoken language identification refers to the task of automatically predicting the spoken language in a given utterance. Conventionally, it is modeled as a speech-based language ide…
Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generation
Motoi Omachi, Brian Yan, Siddharth Dalmia +2
The black-box nature of end-to-end speech translation (E2E ST) systems makes it difficult to understand how source language inputs are being mapped to the target language. To solve…
A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding
Yifan Peng, Siddhant Arora, Yosuke Higuchi +6
Collecting sufficient labeled data for spoken language understanding (SLU) is expensive and time-consuming. Recent studies achieved promising results by using pre-trained models in…
Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models
Siddhant Arora, Siddharth Dalmia, Brian Yan +3
End-to-end spoken language understanding (SLU) systems are gaining popularity over cascaded approaches due to their simplicity and ability to avoid error propagation. However, thes…
CTC Alignments Improve Autoregressive Translation
Brian Yan, Siddharth Dalmia, Yosuke Higuchi +4
Connectionist Temporal Classification (CTC) is a widely used approach for automatic speech recognition (ASR) that performs conditionally independent monotonic alignment. However fo…
Two-Pass Low Latency End-to-End Spoken Language Understanding
Siddhant Arora, Siddharth Dalmia, Xuankai Chang +3
End-to-end (E2E) models are becoming increasingly popular for spoken language understanding (SLU) systems and are beginning to achieve competitive performance to pipeline-based app…
Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding
Yifan Peng, Siddharth Dalmia, Ian Lane +1
Conformer has proven to be effective in many speech processing tasks. It combines the benefits of extracting local dependencies using convolutions and global dependencies using sel…
LegoNN: Building Modular Encoder-Decoder Models
Siddharth Dalmia, Dmytro Okhonko, Mike Lewis +5
State-of-the-art encoder-decoder models (e.g. for machine translation (MT) or automatic speech recognition (ASR)) are constructed and trained end-to-end as an atomic unit. No compo…
FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech
Alexis Conneau, Min Ma, Simran Khanuja +6
We introduce FLEURS, the Few-shot Learning Evaluation of Universal Representations of Speech benchmark. FLEURS is an n-way parallel speech dataset in 102 languages built on top of…
Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorization
Brian Yan, Chunlei Zhang, Meng Yu +6
Conversational bilingual speech encompasses three types of utterances: two purely monolingual types and one intra-sententially code-switched type. In this work, we propose a genera…
ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet
Siddhant Arora, Siddharth Dalmia, Pavel Denisov +10
As Automatic Speech Processing (ASR) systems are getting better, there is an increasing interest of using the ASR output to do downstream Natural Language Processing (NLP) tasks. H…
Differentiable Allophone Graphs for Language-Universal Speech Recognition
Brian Yan, Siddharth Dalmia, David R. Mortensen +2
Building language-universal speech recognition systems entails producing phonological units of spoken sound that can be shared across languages. While speech annotations at the lan…
Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding
Siddhant Arora, Alissa Ostapenko, Vijay Viswanathan +4
Decomposable tasks are complex and comprise of a hierarchy of sub-tasks. Spoken intent prediction, for example, combines automatic speech recognition and natural language understan…
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks
Siddharth Dalmia, Brian Yan, Vikas Raunak +2
End-to-end approaches for sequence tasks are becoming increasingly popular. Yet for complex sequence tasks, like speech translation, systems that cascade several models trained on…
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering
Abhilasha Ravichander, Siddharth Dalmia, Maria Ryskina +3
When Question-Answering (QA) systems are deployed in the real world, users query them through a variety of interfaces, such as speaking to voice assistants, typing questions into a…
Transformer-Transducers for Code-Switched Speech Recognition
Siddharth Dalmia, Yuzong Liu, Srikanth Ronanki +1
We live in a world where 60% of the population can speak two or more languages fluently. Members of these communities constantly switch between languages when having a conversation…
On Long-Tailed Phenomena in Neural Machine Translation
Vikas Raunak, Siddharth Dalmia, Vivek Gupta +1
State-of-the-art Neural Machine Translation (NMT) models struggle with generating low-frequency tokens, tackling which remains a major challenge. The analysis of long-tailed phenom…
Universal Phone Recognition with a Multilingual Allophone System
Xinjian Li, Siddharth Dalmia, Juncheng Li +8
Multilingual models can improve language processing, particularly for low resource situations, by sharing parameters across languages. Multilingual acoustic models, however, genera…
Towards Zero-shot Learning for Automatic Phonemic Transcription
Xinjian Li, Siddharth Dalmia, David R. Mortensen +3
Automatic phonemic transcription tools are useful for low-resource language documentation. However, due to the lack of training sets, only a tiny fraction of languages have phonemi…
Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
Siddharth Dalmia, Abdelrahman Mohamed, Mike Lewis +2
Inspired by modular software design principles of independence, interchangeability, and clarity of interface, we introduce a method for enforcing encoder-decoder modularity in seq2…
SANTLR: Speech Annotation Toolkit for Low Resource Languages
Xinjian Li, Zhong Zhou, Siddharth Dalmia +2
While low resource speech recognition has attracted a lot of attention from the speech community, there are a few tools available to facilitate low resource speech collection. In t…
Multilingual Speech Recognition with Corpus Relatedness Sampling
Xinjian Li, Siddharth Dalmia, Alan W. Black +1
Multilingual acoustic models have been successfully applied to low-resource speech recognition. Most existing works have combined many small corpora together and pretrained a multi…
Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion
Suyoun Kim, Siddharth Dalmia, Florian Metze
We present a novel conversational-context aware end-to-end speech recognizer based on a gated neural network that incorporates conversational-context/word/speech embeddings. Unlike…