Uncertainty aware audiovisual activity recognition using deep Bayesian variational inference

Abstract

Deep neural networks (DNNs) provide state-of-the-art results for a multitude of applications, but the approaches using DNNs for multimodal audiovisual applications do not consider predictive uncertainty associated with individual modalities. Bayesian deep learning methods provide principled confidence and quantify predictive uncertainty. Our contribution in this work is to propose an uncertainty aware multimodal Bayesian fusion framework for activity recognition. We demonstrate a novel approach that combines deterministic and variational layers to scale Bayesian DNNs to deeper architectures. Our experiments using in- and out-of-distribution samples selected from a subset of Moments-in-Time (MiT) dataset show a more reliable confidence measure as compared to the non-Bayesian baseline and the Monte Carlo dropout (MC dropout) approximate Bayesian inference. We also demonstrate the uncertainty estimates obtained from the proposed framework can identify out-of-distribution data on the UCF101 and MiT datasets. In the multimodal setting, the proposed framework improved precision-recall AUC by 10.2% on the subset of MiT dataset as compared to non-Bayesian baseline.

Accepted at ICCV 2019 for Oral presentation

References in corpus (13)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Weight Uncertainty in Neural Networks
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Deep Audio-Visual Speech Recognition
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• A multi-device dataset for urban acoustic scene classification
• Understanding Measures of Uncertainty for Adversarial Example Detection
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
• See, Hear, and Read: Deep Aligned Representations
• Convolutional Neural Networks and x-vector Embedding for DCASE2018 Acoustic Scene Classification Challenge

Cited by in corpus (1)

• Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving