SoundNet: Learning Sound Representations from Unlabeled Video

Abstract

We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild. We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos. Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound. We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge. Our sound representation yields significant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene/object classification. Visualizations suggest some high-level semantics automatically emerge in the sound network, even though it is trained without ground truth labels.

NIPS 2016

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Acoustic Scene Classification

Cited by in corpus (16)

• Audio Spectrogram Representations for Processing with Convolutional Neural Networks
• Comparison of Time-Frequency Representations for Environmental Sound Classification using Convolutional Neural Networks
• See, Hear, and Read: Deep Aligned Representations
• Audio Super Resolution using Neural Networks
• Improving Multi-Modal Learning with Uni-Modal Teachers
• Learning Representations from Audio-Visual Spatial Alignment
• Bandwidth Extension on Raw Audio via Generative Adversarial Networks
• auDeep: Unsupervised Learning of Representations from Audio with Deep Recurrent Neural Networks
• Unsupervised Learning of Semantic Audio Representations
• Deep Multi-Modal Image Correspondence Learning
• Utilizing Domain Knowledge in End-to-End Audio Processing
• Latent Variable Algorithms for Multimodal Learning and Sensor Fusion
• Dual Domain-Adversarial Learning for Audio-Visual Saliency Prediction
• PiNet: Attention Pooling for Graph Classification
• E-Sports Talent Scouting Based on Multimodal Twitch Stream Data
• cvpaper.challenge in 2016: Futuristic Computer Vision through 1,600 Papers Survey