Weakly Labelled AudioSet Tagging with Attention Neural Networks

Abstract

Audio tagging is the task of predicting the presence or absence of sound classes within an audio clip. Previous work in audio tagging focused on relatively small datasets limited to recognising a small number of sound classes. We investigate audio tagging on AudioSet, which is a dataset consisting of over 2 million audio clips and 527 classes. AudioSet is weakly labelled, in that only the presence or absence of sound classes is known for each clip, while the onset and offset times are unknown. To address the weakly-labelled audio tagging problem, we propose attention neural networks as a way to attend the most salient parts of an audio clip. We bridge the connection between attention neural networks and multiple instance learning (MIL) methods, and propose decision-level and feature-level attention neural networks for audio tagging. We investigate attention neural networks modeled by different functions, depths and widths. Experiments on AudioSet show that the feature-level attention neural network achieves a state-of-the-art mean average precision (mAP) of 0.369, outperforming the best multiple instance learning (MIL) method of 0.317 and Google's deep neural network baseline of 0.314. In addition, we discover that the audio tagging performance on AudioSet embedding features has a weak correlation with the number of training samples and the quality of labels of each sound class.

arXiv comment13 pages

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Deep Learning in Neural Networks: An Overview
• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• A Structured Self-attentive Sentence Embedding
• Attention-based Deep Multiple Instance Learning
• Revisiting Multiple Instance Neural Networks
• Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge
• Automatic tagging using deep convolutional neural networks
• Audio Event Detection using Weakly Labeled Data
• A multi-device dataset for urban acoustic scene classification
• General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline
• Automatic Environmental Sound Recognition: Performance versus Computational Cost
• Multi-level Attention Model for Weakly Supervised Audio Classification
• Temporal Attention Model for Neural Machine Translation

Cited by in corpus (16)

• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation
• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
• Audio Retrieval with Natural Language Queries: A Benchmark Study
• PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
• Towards duration robust weakly supervised sound event detection
• Automated Audio Captioning: An Overview of Recent Progress and New Challenges
• Audio Captioning Transformer
• LEAN: Light and Efficient Audio Classification Network
• Secost: Sequential co-supervision for large scale weakly labeled audio event detection
• Sound Event Detection of Weakly Labelled Data with CNN-Transformer and Automatic Threshold Optimization
• Specialized Decision Surface and Disentangled Feature for Weakly-Supervised Polyphonic Sound Event Detection
• Towards Generating Diverse Audio Captions via Adversarial Training
• Active Learning for Sound Event Detection
• A Global-local Attention Framework for Weakly Labelled Audio Tagging
• Advancing Multiple Instance Learning with Attention Modeling for Categorical Speech Emotion Recognition