Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval

Abstract

Speech-based image retrieval has been studied as a proxy for joint representation learning, usually without emphasis on retrieval itself. As such, it is unclear how well speech-based retrieval can work in practice -- both in an absolute sense and versus alternative strategies that combine automatic speech recognition (ASR) with strong text encoders. In this work, we extensively study and expand choices of encoder architectures, training methodology (including unimodal and multimodal pretraining), and other factors. Our experiments cover different types of speech in three datasets: Flickr Audio, Places Audio, and Localized Narratives. Our best model configuration achieves large gains over state of the art, e.g., pushing recall-at-one from 21.8% to 33.2% for Flickr Audio and 27.6% to 53.4% for Places Audio. We also show our best speech-based models can match or exceed cascaded ASR-to-text encoding when speech is spontaneous, accented, or otherwise hard to automatically transcribe.

arXiv commentAccepted to INTERSPEECH 2021

References in corpus (6)

• Representations of language in a model of visually grounded speech signal
• End-to-End Retrieval in Continuous Space
• Language learning using Speech to Image retrieval
• From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
• Learning semantic sentence representations from visually grounded language without lexical knowledge
• Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO

Cited by in corpus (4)

• Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Cascaded Multilingual Audio-Visual Learning from Videos
• Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset