Audio Captioning Transformer
arXiv:2107.09817
Abstract
Audio captioning aims to automatically generate a natural language description of an audio clip. Most captioning models follow an encoder-decoder architecture, where the decoder predicts words based on the audio features extracted by the encoder. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are often used as the audio encoder. However, CNNs can be limited in modelling temporal relationships among the time frames in an audio signal, while RNNs can be limited in modelling the long-range dependencies among the time frames. In this paper, we propose an Audio Captioning Transformer (ACT), which is a full Transformer network based on an encoder-decoder architecture and is totally convolution-free. The proposed method has a better ability to model the global information within an audio signal as well as capture temporal relationships between audio events. We evaluate our model on AudioCaps, which is the largest audio captioning dataset publicly available. Our model shows competitive performance compared to other state-of-the-art approaches.
arXiv comment5 pages, 1 figure
References in corpus (9)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation
- CPTR: Full Transformer Network for Image Captioning
- Weakly Labelled AudioSet Tagging with Attention Neural Networks
- Audio Caption: Listen and Tell
- Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval
- Effects of Word-frequency based Pre- and Post- Processings for Audio Captioning
- WaveTransformer: A Novel Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information
Cited by in corpus (6)
- Audio Retrieval with Natural Language Queries: A Benchmark Study
- Automated Audio Captioning: An Overview of Recent Progress and New Challenges
- CL4AC: A Contrastive Loss for Audio Captioning
- Local Information Assisted Attention-free Decoder for Audio Captioning
- Enhance Temporal Relations in Audio Captioning with Sound Event Detection
- MT3: Multi-Task Multitrack Music Transcription