Continual 3D Convolutional Neural Networks for Real-time Processing of Videos
arXiv:2106.00050 · doi:10.1007/978-3-031-19772-7_22
Abstract
We introduce Continual 3D Convolutional Neural Networks (Co3D CNNs), a new computational formulation of spatio-temporal 3D CNNs, in which videos are processed frame-by-frame rather than by clip. In online tasks demanding frame-wise predictions, Co3D CNNs dispense with the computational redundancies of regular 3D CNNs, namely the repeated convolutions over frames, which appear in overlapping clips. We show that Continual 3D CNNs can reuse preexisting 3D-CNN weights to reduce the per-prediction floating point operations (FLOPs) in proportion to the temporal receptive field while retaining similar memory requirements and accuracy. This is validated with multiple models on Kinetics-400 and Charades with remarkable results: CoX3D models attain state-of-the-art complexity/accuracy trade-offs on Kinetics-400 with 12.1-15.3x reductions of FLOPs and 2.3-3.8% improvements in accuracy compared to regular X3D models while reducing peak memory consumption by up to 48%. Moreover, we investigate the transient response of Co3D CNNs at start-up and perform extensive benchmarks of on-hardware processing characteristics for publicly available 3D CNNs.
arXiv comment22 pages, 9 figures, 7 tables
References in corpus (11)
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- The Kinetics Human Action Video Dataset
- Deep Visual Domain Adaptation: A Survey
- Binarized Neural Networks
- Compressing Neural Networks with the Hashing Trick
- A Short Note about Kinetics-600
- DeepCache: Principled Cache for Mobile Deep Vision
- You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization
- Partial Convolution based Padding
- Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch