MoViNets: Mobile Video Networks for Efficient Video Recognition
arXiv:2103.11511
Abstract
We present Mobile Video Networks (MoViNets), a family of computation and memory efficient video networks that can operate on streaming video for online inference. 3D convolutional neural networks (CNNs) are accurate at video recognition but require large computation and memory budgets and do not support online inference, making them difficult to work on mobile devices. We propose a three-step approach to improve computational efficiency while substantially reducing the peak memory usage of 3D CNNs. First, we design a video network search space and employ neural architecture search to generate efficient and diverse 3D CNN architectures. Second, we introduce the Stream Buffer technique that decouples memory from video clip duration, allowing 3D CNNs to embed arbitrary-length streaming video sequences for both training and inference with a small constant memory footprint. Third, we propose a simple ensembling technique to improve accuracy further without sacrificing efficiency. These three progressive techniques allow MoViNets to achieve state-of-the-art accuracy and efficiency on the Kinetics, Moments in Time, and Charades video action recognition datasets. For instance, MoViNet-A5-Stream achieves the same accuracy as X3D-XL on Kinetics 600 while requiring 80% fewer FLOPs and 65% less memory. Code will be made available at https://github.com/tensorflow/models/tree/master/official/vision.
arXiv commentAccepted to CVPR 2021
References in corpus (23)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Two-Stream Convolutional Networks for Action Recognition in Videos
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Neural Architecture Search with Reinforcement Learning
- The Kinetics Human Action Video Dataset
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Is Space-Time Attention All You Need for Video Understanding?
- Searching for Activation Functions
- AutoAugment: Learning Augmentation Policies from Data
- Efficient Neural Architecture Search via Parameter Sharing
- Born Again Neural Networks
- Towards Good Practices for Very Deep Two-Stream ConvNets
- A Short Note on the Kinetics-700-2020 Human Action Dataset
- A Short Note about Kinetics-600
- ReZero is All You Need: Fast Convergence at Large Depth
- More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation
- Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification
- Learn to cycle: Time-consistent feature discovery for action recognition
- AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
- When Ensembling Smaller Models is More Efficient than Single Large Models
- TAM: Temporal Adaptive Module for Video Recognition
- On Power Laws in Deep Ensembles
- PERF-Net: Pose Empowered RGB-Flow Net