DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs

Abstract

The deployment of Deep Neural Networks (DNNs) on end-nodes at the extreme edge of the Internet-of-Things is a critical enabler to support pervasive Deep Learning-enhanced applications. Low-Cost MCU-based end-nodes have limited on-chip memory and often replace caches with scratchpads, to reduce area overheads and increase energy efficiency -- requiring explicit DMA-based memory transfers between different levels of the memory hierarchy. Mapping modern DNNs on these systems requires aggressive topology-dependent tiling and double-buffering. In this work, we propose DORY (Deployment Oriented to memoRY) - an automatic tool to deploy DNNs on low cost MCUs with typically less than 1MB of on-chip SRAM memory. DORY abstracts tiling as a Constraint Programming (CP) problem: it maximizes L1 memory utilization under the topological constraints imposed by each DNN layer. Then, it generates ANSI C code to orchestrate off- and on-chip transfers and computation phases. Furthermore, to maximize speed, DORY augments the CP formulation with heuristics promoting performance-effective tile sizes. As a case study for DORY, we target GreenWaves Technologies GAP8, one of the most advanced parallel ultra-low power MCU-class devices on the market. On this device, DORY achieves up to 2.5x better MAC/cycle than the GreenWaves proprietary software solution and 18.1x better than the state-of-the-art result on an STM32-F746 MCU on single layers. Using our tool, GAP-8 can perform end-to-end inference of a 1.0-MobileNet-128 network consuming just 63 pJ/MAC on average @ 4.3 fps - 15.4x better than an STM32-F746. We release all our developments - the DORY framework, the optimized backend kernels, and the related heuristics - as open-source software.

arXiv comment14 pages, 12 figures, 4 tables, 2 listings. Accepted for publication in IEEE Transactions on Computers (https://ieeexplore.ieee.org/document/9381618)

References in corpus (6)

• Machine learning for Internet of Things data analysis: A survey
• Interstellar: Using Halide's Scheduling Language to Analyze DNN Accelerators
• Hardware for Machine Learning: Challenges and Opportunities
• A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones
• PULP-NN: Accelerating Quantized Neural Networks on Parallel Ultra-Low-Power RISC-V Processors
• An IoT Endpoint System-on-Chip for Secure and Energy-Efficient Near-Sensor Analytics

Cited by in corpus (24)

• Machine Learning for Microcontroller-Class Hardware: A Review
• Tiny Machine Learning: Progress and Futures
• Vega: A 10-Core SoC for IoT End-Nodes with DNN Acceleration and Cognitive Wake-Up From MRAM-Based State-Retentive Sleep Mode
• A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays
• Lightweight Neural Architecture Search for Temporal Convolutional Networks at the Edge
• GVSoC: A Highly Configurable, Fast and Accurate Full-Platform Simulator for RISC-V based IoT Processors
• A Tiny Transformer for Low-Power Arrhythmia Classification on Microcontrollers
• Marsellus: A Heterogeneous RISC-V AI-IoT End-Node SoC with 2-to-8b DNN Acceleration and 30%-Boost Adaptive Body Biasing
• DARKSIDE: A Heterogeneous RISC-V Compute Cluster for Extreme-Edge On-Chip DNN Inference and Training
• Improving Autonomous Nano-drones Performance via Automated End-to-End Optimization and Deployment of DNNs
• Dynamic Decision Tree Ensembles for Energy-Efficient Inference on IoT Edge Nodes
• Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices
• Flexible and Fully Quantized Ultra-Lightweight TinyissimoYOLO for Ultra-Low-Power Edge Systems
• Mixed-precision Neural Networks on RISC-V Cores: ISA extensions for Multi-Pumped Soft SIMD Operations
• HULK-V: a Heterogeneous Ultra-low-power Linux capable RISC-V SoC
• A Heterogeneous RISC-V based SoC for Secure Nano-UAV Navigation
• Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes
• Ultra-low Power Deep Learning-based Monocular Relative Localization Onboard Nano-quadrotors
• HTVM: Efficient Neural Network Deployment On Heterogeneous TinyML Platforms
• Deep Neural Network Architecture Search for Accurate Visual Pose Estimation aboard Nano-UAVs
• End-to-end 100-TOPS/W Inference With Analog In-Memory Computing: Are We There Yet?
• An Ultra-Low Power Wearable BMI System with Continual Learning Capabilities
• OpTC -- A Toolchain for Deployment of Neural Networks on AURIX TC3xx Microcontrollers
• MEDEA: A Design-Time Multi-Objective Manager for Energy-Efficient DNN Inference on Heterogeneous Ultra-Low Power Platforms