Can audio-visual integration strengthen robustness under multimodal attacks?

Abstract

In this paper, we propose to make a systematic study on machines multisensory perception under attacks. We use the audio-visual event recognition task against multimodal adversarial attacks as a proxy to investigate the robustness of audio-visual learning. We attack audio, visual, and both modalities to explore whether audio-visual integration still strengthens perception and how different fusion mechanisms affect the robustness of audio-visual models. For interpreting the multimodal interactions under attacks, we learn a weakly-supervised sound source visual localization model to localize sounding regions in videos. To mitigate multimodal attacks, we propose an audio-visual defense approach based on an audio-visual dissimilarity constraint and external feature memory banks. Extensive experiments demonstrate that audio-visual models are susceptible to multimodal adversarial attacks; audio-visual integration could decrease the model robustness rather than strengthen under multimodal attacks; even a weakly-supervised sound source visual localization model can be successfully fooled; our defense method can improve the invulnerability of audio-visual networks without significantly sacrificing clean model performance.

arXiv commentCVPR 2021

References in corpus (16)

• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks
• Ensemble Adversarial Training: Attacks and Defenses
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Countering Adversarial Images using Input Transformations
• Distributional Smoothing with Virtual Adversarial Training
• Mitigating Adversarial Effects Through Randomization
• Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition
• Audiovisual SlowFast Networks for Video Recognition
• Characterizing Audio Adversarial Examples Using Temporal Dependency
• Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
• Efficient Large-Scale Multi-Modal Classification
• Listen to Look: Action Recognition by Previewing Audio
• VisualEchoes: Spatial Image Representation Learning through Echolocation
• Towards Resistant Audio Adversarial Examples
• Detecting Adversarial Attacks On Audiovisual Speech Recognition