LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition

Abstract

This paper presents LiteEval, a simple yet effective coarse-to-fine framework for resource efficient video recognition, suitable for both online and offline scenarios. Exploiting decent yet computationally efficient features derived at a coarse scale with a lightweight CNN model, LiteEval dynamically decides on-the-fly whether to compute more powerful features for incoming video frames at a finer scale to obtain more details. This is achieved by a coarse LSTM and a fine LSTM operating cooperatively, as well as a conditional gating module to learn when to allocate more computation. Extensive experiments are conducted on two large-scale video benchmarks, FCVID and ActivityNet, and the results demonstrate LiteEval requires substantially less computation while offering excellent classification accuracy for both online and offline predictions.

NeurIPS 2019

Cited by in corpus (9)

• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• Dynamic Neural Networks: A Survey
• AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
• AR-Net: Adaptive Frame Resolution for Efficient Action Recognition
• Adaptive Focus for Efficient Video Recognition
• Coarse-to-Fine Curriculum Learning
• Dynamic Network Quantization for Efficient Video Inference
• Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints
• Skimming and Scanning for Untrimmed Video Action Recognition