Audio Caption: Listen and Tell

Abstract

Increasing amount of research has shed light on machine perception of audio events, most of which concerns detection and classification tasks. However, human-like perception of audio scenes involves not only detecting and classifying audio sounds, but also summarizing the relationship between different audio events. Comparable research such as image caption has been conducted, yet the audio field is still quite barren. This paper introduces a manually-annotated dataset for audio caption. The purpose is to automatically generate natural sentences for audio scene description and to bridge the gap between machine perception of audio and image. The whole dataset is labelled in Mandarin and we also include translated English annotations. A baseline encoder-decoder model is provided for both English and Mandarin. Similar BLEU scores are derived for both languages: our model can generate understandable and data-related captions based on the dataset.

accepted by ICASSP2019

References in corpus (1)

• Automatic tagging using deep convolutional neural networks

Cited by in corpus (19)

• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
• Audio Retrieval with Natural Language Queries: A Benchmark Study
• Automated Audio Captioning: An Overview of Recent Progress and New Challenges
• Audio Captioning Transformer
• MusCaps: Generating Captions for Music Audio
• The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimation
• Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval
• Multi-task Regularization Based on Infrequent Classes for Audio Captioning
• Effects of Word-frequency based Pre- and Post- Processings for Audio Captioning
• Improving the Performance of Automated Audio Captioning via Integrating the Acoustic and Semantic Information
• Enhance Temporal Relations in Audio Captioning with Sound Event Detection
• Audio Captioning using Gated Recurrent Units
• WaveTransformer: A Novel Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information
• Temporal Sub-sampling of Audio Feature Sequences for Automated Audio Captioning
• Audio-Language Datasets of Scenes and Events: A Survey
• Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning
• A Transformer-based Audio Captioning Model with Keyword Estimation
• Text-to-Audio Grounding: Building Correspondence Between Captions and Sound Events
• Audio Caption in a Car Setting with a Sentence-Level Loss