Enhance Temporal Relations in Audio Captioning with Sound Event Detection

Abstract

Automated audio captioning aims at generating natural language descriptions for given audio clips, not only detecting and classifying sounds, but also summarizing the relationships between audio events. Recent research advances in audio captioning have introduced additional guidance to improve the accuracy of audio events in generated sentences. However, temporal relations between audio events have received little attention while revealing complex relations is a key component in summarizing audio content. Therefore, this paper aims to better capture temporal relationships in caption generation with sound event detection (SED), a task that locates events' timestamps. We investigate the best approach to integrate temporal information in a captioning model and propose a temporal tag system to transform the timestamps into comprehensible relations. Results evaluated by the proposed temporal metrics suggest that great improvement is achieved in terms of temporal relation generation.

Interspeech 2023

References in corpus (11)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Audio Caption: Listen and Tell
• Towards duration robust weakly supervised sound event detection
• Audio Captioning Transformer
• The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimation
• Multi-task Regularization Based on Infrequent Classes for Audio Captioning
• CL4AC: A Contrastive Loss for Audio Captioning
• Improving the Performance of Automated Audio Captioning via Integrating the Acoustic and Semantic Information
• Audio Captioning Using Sound Event Detection
• Investigating Local and Global Information for Automated Audio Captioning with Transfer Learning
• Can Audio Captions Be Evaluated with Image Caption Metrics?