Audio Retrieval with Natural Language Queries: A Benchmark Study

Abstract

The objectives of this work are cross-modal text-audio and audio-text retrieval, in which the goal is to retrieve the audio content from a pool of candidates that best matches a given written description and vice versa. Text-audio retrieval enables users to search large databases through an intuitive interface: they simply issue free-form natural language descriptions of the sound they would like to hear. To study the tasks of text-audio and audio-text retrieval, which have received limited attention in the existing literature, we introduce three challenging new benchmarks. We first construct text-audio and audio-text retrieval benchmarks from the AudioCaps and Clotho audio captioning datasets. Additionally, we introduce the SoundDescs benchmark, which consists of paired audio and natural language descriptions for a diverse collection of sounds that are complementary to those found in AudioCaps and Clotho. We employ these three benchmarks to establish baselines for cross-modal text-audio and audio-text retrieval, where we demonstrate the benefits of pre-training on diverse audio tasks. We hope that our benchmarks will inspire further research into audio retrieval with free-form text queries. Code, audio features for all datasets used, and the SoundDescs dataset are publicly available at https://github.com/akoepke/audio-retrieval-benchmark.

arXiv commentSubmitted to Transactions on Multimedia. arXiv admin note: substantial text overlap with arXiv:2105.02192

References in corpus (15)

• Efficient Estimation of Word Representations in Vector Space
• YouTube-8M: A Large-Scale Video Classification Benchmark
• On the Variance of the Adaptive Learning Rate and Beyond
• Lookahead Optimizer: k steps forward, 1 step back
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• Predicting Visual Features from Text for Image and Video Caption Retrieval
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• Weakly Labelled AudioSet Tagging with Attention Neural Networks
• Multi-level Attention Model for Weakly Supervised Audio Classification
• Audio Caption: Listen and Tell
• Word2VisualVec: Image and Video to Sentence Matching by Visual Feature Prediction
• Audio Captioning Transformer
• Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval
• CL4AC: A Contrastive Loss for Audio Captioning

Cited by in corpus (10)

• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
• Automated Audio Captioning: An Overview of Recent Progress and New Challenges
• SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description
• Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training
• Data leakage in cross-modal retrieval training: A case study
• When the Music Stops: Tip-of-the-Tongue Retrieval for Music
• Audio-Language Datasets of Scenes and Events: A Survey
• NAAQA: A Neural Architecture for Acoustic Question Answering
• AudioNet: Supervised Deep Hashing for Retrieval of Similar Audio Events
• Distance Sampling-based Paraphraser Leveraging ChatGPT for Text Data Manipulation