SoundNet: Learning Sound Representations from Unlabeled Video

Abstract

We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild. We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos. Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound. We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge. Our sound representation yields significant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene/object classification. Visualizations suggest some high-level semantics automatically emerge in the sound network, even though it is trained without ground truth labels.

arXiv commentNIPS 2016

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• YFCC100M: The New Data in Multimedia Research
• Do Deep Nets Really Need to be Deep?
• Acoustic Scene Classification
• Cross Modal Distillation for Supervision Transfer

Cited by in corpus (45)

• Machine learning in acoustics: theory and applications
• Multi-task Self-Supervised Learning for Human Activity Detection
• Learning from Between-class Examples for Deep Sound Recognition
• Audio Spectrogram Representations for Processing with Convolutional Neural Networks
• Transfer learning for music classification and regression tasks
• Comparison of Time-Frequency Representations for Environmental Sound Classification using Convolutional Neural Networks
• General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline
• Rethinking CNN Models for Audio Classification
• Audio-Based Activities of Daily Living (ADL) Recognition with Large-Scale Acoustic Embeddings from Online Videos
• A Tutorial on Deep Learning for Music Information Retrieval
• Audio style transfer
• See, Hear, and Read: Deep Aligned Representations
• DEPA: Self-Supervised Audio Embedding for Depression Detection
• Time-Contrastive Networks: Self-Supervised Learning from Video
• Look, Listen and Learn
• Audio Super Resolution using Neural Networks
• DCASE 2018 Challenge Surrey Cross-Task convolutional neural network baseline
• Improving Multi-Modal Learning with Uni-Modal Teachers
• Learning Representations from Audio-Visual Spatial Alignment
• On Learning Associations of Faces and Voices
• Bandwidth Extension on Raw Audio via Generative Adversarial Networks
• Temporal-Spatial Feature Pyramid for Video Saliency Detection
• Brain-mediated Transfer Learning of Convolutional Neural Networks
• auDeep: Unsupervised Learning of Representations from Audio with Deep Recurrent Neural Networks
• An Attempt towards Interpretable Audio-Visual Video Captioning
• Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
• Weakly Supervised Representation Learning for Unsynchronized Audio-Visual Events
• Speech-Based Visual Question Answering
• Sample-level CNN Architectures for Music Auto-tagging Using Raw Waveforms
• Unsupervised Learning of Semantic Audio Representations
• Deep Convolutional Neural Network with Mixup for Environmental Sound Classification
• One Deep Music Representation to Rule Them All? : A comparative analysis of different representation learning strategies
• Utilizing Domain Knowledge in End-to-End Audio Processing
• Deep Multi-Modal Image Correspondence Learning
• Towards Audio to Scene Image Synthesis using Generative Adversarial Network
• Competitive Learning Enriches Learning Representation and Accelerates the Fine-tuning of CNNs
• A Deeper Look at 3D Shape Classifiers
• Sample Dropout for Audio Scene Classification Using Multi-Scale Dense Connected Convolutional Neural Network
• Dual Domain-Adversarial Learning for Audio-Visual Saliency Prediction
• Latent Variable Algorithms for Multimodal Learning and Sensor Fusion
• Pseudo-labels for Supervised Learning on Dynamic Vision Sensor Data, Applied to Object Detection under Ego-motion
• E-Sports Talent Scouting Based on Multimodal Twitch Stream Data
• cvpaper.challenge in 2016: Futuristic Computer Vision through 1,600 Papers Survey
• PiNet: Attention Pooling for Graph Classification
• A Light-Weight Multimodal Framework for Improved Environmental Audio Tagging