Reducing Inference Latency with Concurrent Architectures for Image Recognition

Abstract

Satisfying the high computation demand of modern deep learning architectures is challenging for achieving low inference latency. The current approaches in decreasing latency only increase parallelism within a layer. This is because architectures typically capture a single-chain dependency pattern that prevents efficient distribution with a higher concurrency (i.e., simultaneous execution of one inference among devices). Such single-chain dependencies are so widespread that even implicitly biases recent neural architecture search (NAS) studies. In this visionary paper, we draw attention to an entirely new space of NAS that relaxes the single-chain dependency to provide higher concurrency and distribution opportunities. To quantitatively compare these architectures, we propose a score that encapsulates crucial metrics such as communication, concurrency, and load balancing. Additionally, we propose a new generator and transformation block that consistently deliver superior architectures compared to current state-of-the-art methods. Finally, our preliminary results show that these new architectures reduce the inference latency and deserve more attention.

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Statistical mechanics of complex networks
• Neural Architecture Search with Reinforcement Learning
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Renormalization group analysis of the small-world network model
• Compressing Deep Convolutional Networks using Vector Quantization
• Ternary Weight Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Training deep neural networks with low precision multiplications
• A Survey on Neural Architecture Search
• Musical Chair: Efficient Real-Time Recognition Using Collaborative IoT Devices