WaveTransformer: A Novel Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information

Abstract

Automated audio captioning (AAC) is a novel task, where a method takes as an input an audio sample and outputs a textual description (i.e. a caption) of its contents. Most AAC methods are adapted from from image captioning of machine translation fields. In this work we present a novel AAC novel method, explicitly focused on the exploitation of the temporal and time-frequency patterns in audio. We employ three learnable processes for audio encoding, two for extracting the local and temporal information, and one to merge the output of the previous two processes. To generate the caption, we employ the widely used Transformer decoder. We assess our method utilizing the freely available splits of Clotho dataset. Our results increase previously reported highest SPIDEr to 17.3, from 16.2.

arXiv commentSubmitted for review at ICASSP2021

References in corpus (8)

• WaveNet: A Generative Model for Raw Audio
• Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Twin Networks: Matching the Future for Sequence Generation
• Audio Caption: Listen and Tell
• Multi-task Regularization Based on Infrequent Classes for Audio Captioning
• Effects of Word-frequency based Pre- and Post- Processings for Audio Captioning
• Temporal Sub-sampling of Audio Feature Sequences for Automated Audio Captioning

Cited by in corpus (5)

• Automated Audio Captioning: An Overview of Recent Progress and New Challenges
• Audio Captioning Transformer
• MusCaps: Generating Captions for Music Audio
• An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning
• CL4AC: A Contrastive Loss for Audio Captioning