AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures

Abstract

Learning to represent videos is a very challenging task both algorithmically and computationally. Standard video CNN architectures have been designed by directly extending architectures devised for image understanding to include the time dimension, using modules such as 3D convolutions, or by using two-stream design to capture both appearance and motion in videos. We interpret a video CNN as a collection of multi-stream convolutional blocks connected to each other, and propose the approach of automatically finding neural architectures with better connectivity and spatio-temporal interactions for video understanding. This is done by evolving a population of overly-connected architectures guided by connection weight learning. Architectures combining representations that abstract different input types (i.e., RGB and optical flow) at multiple temporal resolutions are searched for, allowing different types or sources of information to interact with each other. Our method, referred to as AssembleNet, outperforms prior approaches on public video datasets, in some cases by a great margin. We obtain 58.6% mAP on Charades and 34.27% accuracy on Moments-in-Time.

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Neural Architecture Search with Reinforcement Learning
• Spatiotemporal Residual Networks for Video Action Recognition
• SlowFast Networks for Video Recognition
• Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
• Exploring Randomly Wired Neural Networks for Image Recognition
• Long-Term Feature Banks for Detailed Video Understanding
• Connectivity Learning in Multi-Branch Networks

Cited by in corpus (11)

• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Attention Bottlenecks for Multimodal Fusion
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Evaluating Efficient Performance Estimators of Neural Architectures
• MoViNets: Mobile Video Networks for Efficient Video Recognition
• PERF-Net: Pose Empowered RGB-Flow Net
• Learning Temporally Invariant and Localizable Features via Data Augmentation for Video Recognition
• Spatiotemporal Fusion in 3D CNNs: A Probabilistic View
• Multi-Label Activity Recognition using Activity-specific Features and Activity Correlations
• Recent Progress in Appearance-based Action Recognition
• Efficient Modelling Across Time of Human Actions and Interactions