2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition

Abstract

3D convolutional networks are prevalent for video recognition. While achieving excellent recognition performance on standard benchmarks, they operate on a sequence of frames with 3D convolutions and thus are computationally demanding. Exploiting large variations among different videos, we introduce Ada3D, a conditional computation framework that learns instance-specific 3D usage policies to determine frames and convolution layers to be used in a 3D network. These policies are derived with a two-head lightweight selection network conditioned on each input video clip. Then, only frames and convolutions that are selected by the selection network are used in the 3D model to generate predictions. The selection network is optimized with policy gradient methods to maximize a reward that encourages making correct predictions with limited computation. We conduct experiments on three video recognition benchmarks and demonstrate that our method achieves similar accuracies to state-of-the-art 3D models while requiring 20%-50% less computation across different datasets. We also show that learned policies are transferable and Ada3D is compatible to different backbones and modern clip selection approaches. Our qualitative analysis indicates that our method allocates fewer 3D convolutions and frames for "static" inputs, yet uses more for motion-intensive clips.

arXiv commentCVPR 2021

References in corpus (7)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Two-Stream Convolutional Networks for Action Recognition in Videos
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Dynamic Sampling Networks for Efficient Action Recognition in Videos
• Batch-Shaping for Learning Conditional Channel Gated Networks

Cited by in corpus (1)

• Adaptive Focus for Efficient Video Recognition