187 citations · 224 across the 14 of their papers we have counts for
23 papers
TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Freeman Jiang, Ramon Sanabria, Soham Deshmukh +16
Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor. However, turn-taking evaluation remains limited du…
Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems
Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung +3
Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval sy…
Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognition
Alexandra Saliba, Yuanchao Li, Ramon Sanabria +1
The efficacy of self-supervised speech models has been validated, yet the optimal utilization of their representations remains challenging across diverse tasks. In this study, we d…
Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Pooling
Ramon Sanabria, Ondrej Klejch, Hao Tang +1
Acoustic word embeddings are typically created by training a pooling function using pairs of word-like units. For unsupervised systems, these are mined using k-nearest neighbor (KN…
The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR
Ramon Sanabria, Nikolay Bogoychev, Nina Markl +3
English is the most widely spoken language in the world, used daily by millions of people as a first or second language in many different contexts. As a result, there are many vari…
Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models
Ramon Sanabria, Hao Tang, Sharon Goldwater
Given the strong results of self-supervised models on various tasks, there have been surprisingly few studies exploring self-supervised representations for acoustic word embeddings…
Measuring the Impact of Individual Domain Factors in Self-Supervised Pre-Training
Ramon Sanabria, Wei-Ning Hsu, Alexei Baevski +1
Human speech data comprises a rich set of domain factors such as accent, syntactic and semantic variety, or acoustic environment. Previous work explores the effect of domain mismat…
On the Difficulty of Segmenting Words with Attention
Ramon Sanabria, Hao Tang, Sharon Goldwater
Word segmentation, the problem of finding word boundaries in speech, is of interest for a range of tasks. Previous papers have suggested that for sequence-to-sequence models traine…
Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval
Ramon Sanabria, Austin Waters, Jason Baldridge
Speech-based image retrieval has been studied as a proxy for joint representation learning, usually without emphasis on retrieval itself. As such, it is unclear how well speech-bas…
Multimodal Speech Recognition with Unstructured Audio Masking
Tejas Srinivasan, Ramon Sanabria, Florian Metze +1
Visual context has been shown to be useful for automatic speech recognition (ASR) systems when the speech signal is noisy or corrupted. Previous work, however, has only demonstrate…
Fine-Grained Grounding for Multimodal Speech Recognition
Tejas Srinivasan, Ramon Sanabria, Florian Metze +1
Multimodal automatic speech recognition systems integrate information from images to improve speech recognition quality, by grounding the speech in the visual context. While visual…
Looking Enhances Listening: Recovering Missing Speech Using Images
Tejas Srinivasan, Ramon Sanabria, Florian Metze
Speech is understood better by using visual context; for this reason, there have been many attempts to use images to adapt automatic speech recognition (ASR) systems. Current work,…
Multitask Learning For Different Subword Segmentations In Neural Machine Translation
Tejas Srinivasan, Ramon Sanabria, Florian Metze
In Neural Machine Translation (NMT) the usage of subwords and characters as source and target units offers a simple and flexible solution for translation of rare and unseen words.…
Grounding Object Detections With Transcriptions
Yasufumi Moriya, Ramon Sanabria, Florian Metze +1
A vast amount of audio-visual data is available on the Internet thanks to video streaming services, to which users upload their content. However, there are difficulties in exploiti…
Analyzing Utility of Visual Context in Multimodal Speech Recognition Under Noisy Conditions
Tejas Srinivasan, Ramon Sanabria, Florian Metze
Multimodal learning allows us to leverage information from multiple sources (visual, acoustic and text), similar to our experience of the real world. However, it is currently uncle…
How2: A Large-scale Dataset for Multimodal Language Understanding
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar +4
In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We also present integrated sequen…
Multimodal Grounding for Sequence-to-Sequence Speech Recognition
Ozan Caglayan, Ramon Sanabria, Shruti Palaskar +2
Humans are capable of processing speech by making use of multiple sensory modalities. For example, the environment where a conversation takes place generally provides semantic and/…
Hierarchical Multi Task Learning With CTC
Ramon Sanabria, Florian Metze
In Automatic Speech Recognition it is still challenging to learn useful intermediate representations when using high-level (or abstract) target units such as words. For that reason…
End-to-End Multimodal Speech Recognition
Shruti Palaskar, Ramon Sanabria, Florian Metze
Transcription or sub-titling of open-domain videos is still a challenging domain for Automatic Speech Recognition (ASR) due to the data's challenging acoustics, variable signal pro…
Sequence-based Multi-lingual Low Resource Speech Recognition
Siddharth Dalmia, Ramon Sanabria, Florian Metze +1
Techniques for multi-lingual and cross-lingual speech recognition can help in low resource scenarios, to bootstrap systems and enable analysis of new languages and domains. End-to-…
Subword and Crossword Units for CTC Acoustic Models
Thomas Zenkel, Ramon Sanabria, Florian Metze +1
This paper proposes a novel approach to create an unit set for CTC based speech recognition systems. By using Byte Pair Encoding we learn an unit set of an arbitrary size on a give…
Comparison of Decoding Strategies for CTC Acoustic Models
Thomas Zenkel, Ramon Sanabria, Florian Metze +4
Connectionist Temporal Classification has recently attracted a lot of interest as it offers an elegant approach to building acoustic models (AMs) for speech recognition. The CTC lo…
Robust end-to-end deep audiovisual speech recognition
Ramon Sanabria, Florian Metze, Fernando De La Torre
Speech is one of the most effective ways of communication among humans. Even though audio is the most common way of transmitting speech, very important information can be found in…