Recurrent Residual Module for Fast Inference in Videos

Abstract

Deep convolutional neural networks (CNNs) have made impressive progress in many video recognition tasks such as video pose estimation and video object detection. However, CNN inference on video is computationally expensive due to processing dense frames individually. In this work, we propose a framework called Recurrent Residual Module (RRM) to accelerate the CNN inference for video recognition tasks. This framework has a novel design of using the similarity of the intermediate feature maps of two consecutive frames, to largely reduce the redundant computation. One unique property of the proposed method compared to previous work is that feature maps of each frame are precisely computed. The experiments show that, while maintaining the similar recognition performance, our RRM yields averagely 2x acceleration on the commonly used CNNs such as AlexNet, ResNet, deep compression model (thus 8-12x faster than the original dense models using the efficient inference engine), and impressively 9x acceleration on some binary networks such as XNOR-Nets (thus 500x faster than the original model). We further verify the effectiveness of the RRM on speeding up CNNs for video pose estimation and video object detection.

arXiv commentTo appear in CVPR 2018

References in corpus (27)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Learning Structured Sparsity in Deep Neural Networks
• YOLO9000: Better, Faster, Stronger
• Channel Pruning for Accelerating Very Deep Neural Networks
• Provable Bounds for Learning Some Deep Representations
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Memory Bounded Deep Convolutional Networks
• The Power of Sparsity in Convolutional Neural Networks
• Exploiting Image-trained CNN Architectures for Unconstrained Video Classification
• RMPE: Regional Multi-person Pose Estimation
• NoScope: Optimizing Neural Network Queries over Video at Scale
• Predictive-Corrective Networks for Action Detection
• More is Less: A More Complicated Network with Less Inference Complexity
• Temporal Relational Reasoning in Videos
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• Performance Guaranteed Network Acceleration via High-Order Residual Quantization
• Deep Fried Convnets