Audio Caption in a Car Setting with a Sentence-Level Loss
arXiv:1905.13448
Abstract
Captioning has attracted much attention in image and video understanding while a small amount of work examines audio captioning. This paper contributes a Mandarin-annotated dataset for audio captioning within a car scene. A sentence-level loss is proposed to be used in tandem with a GRU encoder-decoder model to generate captions with higher semantic similarity to human annotations. We evaluate the model on the newly-proposed Car dataset, a previously published Mandarin Hospital dataset and the Joint dataset, indicating its generalization capability across different scenes. An improvement in all metrics can be observed, including classical natural language generation (NLG) metrics, sentence richness and human evaluation ratings. However, though detailed audio captions can now be automatically generated, human annotations still outperform model captions on many aspects.
References in corpus (6)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Distributed Representations of Sentences and Documents
- Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
- Audio Caption: Listen and Tell
- The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimation