RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference

Abstract

Standard Convolutional Neural Networks (CNNs) designed for computer vision tasks tend to have large intermediate activation maps. These require large working memory and are thus unsuitable for deployment on resource-constrained devices typically used for inference on the edge. Aggressively downsampling the images via pooling or strided convolutions can address the problem but leads to a significant decrease in accuracy due to gross aggregation of the feature map by standard pooling operators. In this paper, we introduce RNNPool, a novel pooling operator based on Recurrent Neural Networks (RNNs), that efficiently aggregates features over large patches of an image and rapidly downsamples activation maps. Empirical evaluation indicates that an RNNPool layer can effectively replace multiple blocks in a variety of architectures such as MobileNets, DenseNet when applied to standard vision tasks like image classification and face detection. That is, RNNPool can significantly decrease computational complexity and peak memory usage for inference while retaining comparable accuracy. We use RNNPool with the standard S3FD architecture to construct a face detection method that achieves state-of-the-art MAP for tiny ARM Cortex-M4 class microcontrollers with under 256 KB of RAM. Code is released at https://github.com/Microsoft/EdgeML.

arXiv comment25 pages, 8 figures. Published at Advances in Neural Information Processing Systems (NeurIPS) 2020

References in corpus (11)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• The State of Sparsity in Deep Neural Networks
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Visual Wake Words Dataset
• MEC: Memory-efficient Convolution for Deep Neural Network
• LFFD: A Light and Fast Face Detector for Edge Devices
• EXTD: Extremely Tiny Face Detector via Iterative Filter Reuse

Cited by in corpus (6)

• Machine Learning for Microcontroller-Class Hardware: A Review
• Tiny Machine Learning: Progress and Futures
• Going Deeper Into Face Detection: A Survey
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• Implicit Bias of Linear RNNs