WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research

Abstract

The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years. However, researchers face challenges due to the costly and time-consuming collection process of existing audio-language datasets, which are limited in size. To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. We sourced audio clips and their raw descriptions from web sources and a sound event detection dataset. However, the online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. To overcome this issue, we propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically. We conduct a comprehensive analysis of the characteristics of WavCaps dataset and evaluate it on multiple downstream audio-language multimodal learning tasks. The systems trained on WavCaps outperform previous state-of-the-art (SOTA) models by a significant margin. Our aspiration is for the WavCaps dataset we have proposed to facilitate research in audio-language multimodal learning and demonstrate the potential of utilizing ChatGPT to enhance academic research. Our dataset and codes are available at https://github.com/XinhaoMei/WavCaps.

Accepted to TASLP

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Acoustic Scene Classification
• nocaps: novel object captioning at scale
• Sound Event Detection: A Tutorial
• MusicLM: Generating Music From Text
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• Unsupervised Feature Learning Based on Deep Models for Environmental Audio Tagging
• Weakly Labelled AudioSet Tagging with Attention Neural Networks
• Audio Retrieval with Natural Language Queries: A Benchmark Study
• Audio Caption: Listen and Tell
• Automated Audio Captioning: An Overview of Recent Progress and New Challenges
• End-to-End Audio Strikes Back: Boosting Augmentations Towards An Efficient Audio Classification Network
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation
• ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst

Cited by in corpus (4)

• A Survey on Multimodal Large Language Models
• SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound
• Acoustic Prompt Tuning: Empowering Large Language Models with Audition Capabilities
• Audio-Language Datasets of Scenes and Events: A Survey