CL4AC: A Contrastive Loss for Audio Captioning
arXiv:2107.09990
Abstract
Automated Audio captioning (AAC) is a cross-modal translation task that aims to use natural language to describe the content of an audio clip. As shown in the submissions received for Task 6 of the DCASE 2021 Challenges, this problem has received increasing interest in the community. The existing AAC systems are usually based on an encoder-decoder architecture, where the audio signal is encoded into a latent representation, and aligned with its corresponding text descriptions, then a decoder is used to generate the captions. However, training of an AAC system often encounters the problem of data scarcity, which may lead to inaccurate representation and audio-text alignment. To address this problem, we propose a novel encoder-decoder framework called Contrastive Loss for Audio Captioning (CL4AC). In CL4AC, the self-supervision signals derived from the original audio-text paired data are used to exploit the correspondences between audio and texts by contrasting samples, which can improve the quality of latent representation and the alignment between audio and texts, while trained with limited data. Experiments are performed on the Clotho dataset to show the effectiveness of our proposed approach.
arXiv commentThe first two authors contributed equally, 5 pages, 3 figures, accepted by DCASE2021 Workshop
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Efficient Estimation of Word Representations in Vector Space
- Distributed Representations of Words and Phrases and their Compositionality
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
- Audio Captioning Transformer
- Token-Level Supervised Contrastive Learning for Punctuation Restoration
- An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning
- Effects of Word-frequency based Pre- and Post- Processings for Audio Captioning
- WaveTransformer: A Novel Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information