LCP: A Low-Communication Parallelization Method for Fast Neural Network Inference in Image Recognition

Abstract

Deep neural networks (DNNs) have inspired new studies in myriad edge applications with robots, autonomous agents, and Internet-of-things (IoT) devices. However, performing inference of DNNs in the edge is still a severe challenge, mainly because of the contradiction between the intensive resource requirements of DNNs and the tight resource availability in several edge domains. Further, as communication is costly, taking advantage of other available edge devices by using data- or model-parallelism methods is not an effective solution. To benefit from available compute resources with low communication overhead, we propose the first DNN parallelization method for reducing the communication overhead in a distributed system. We propose a low-communication parallelization (LCP) method in which models consist of several almost-independent and narrow branches. LCP offers close-to-minimum communication overhead with better distribution and parallelization opportunities while significantly reducing memory footprint and computation compared to data- and model-parallelism methods. We deploy LCP models on three distributed systems: AWS instances, Raspberry Pis, and PYNQ boards. We also evaluate the performance of LCP models on a customized hardware (tailored for low latency) implemented on a small edge FPGA and as a 16mW 0.107mm2 ASIC @7nm chip. LCP models achieve a maximum and average speedups of 56x and 7x, compared to the originals, which could be improved by up to an average speedup of 33x by incorporating common optimizations such as pruning and quantization.

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Neural Architecture Search with Reinforcement Learning
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• cuDNN: Efficient Primitives for Deep Learning
• Compressing Deep Convolutional Networks using Vector Quantization
• One weird trick for parallelizing convolutional neural networks
• Ternary Weight Networks
• Fixed Point Quantization of Deep Convolutional Networks
• From Perception to Decision: A Data-driven Approach to End-to-end Motion Planning for Autonomous Ground Robots
• Training deep neural networks with low precision multiplications
• Collaborative Large-Scale Dense 3D Reconstruction with Online Inter-Agent Pose Optimisation
• Musical Chair: Efficient Real-Time Recognition Using Collaborative IoT Devices
• Collaborative Execution of Deep Neural Networks on Internet of Things Devices

Cited by in corpus (1)

• DEFER: Distributed Edge Inference for Deep Neural Networks