A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones

Abstract

Fully-autonomous miniaturized robots (e.g., drones), with artificial intelligence (AI) based visual navigation capabilities are extremely challenging drivers of Internet-of-Things edge intelligence capabilities. Visual navigation based on AI approaches, such as deep neural networks (DNNs) are becoming pervasive for standard-size drones, but are considered out of reach for nanodrones with size of a few cm${}^2$. In this work, we present the first (to the best of our knowledge) demonstration of a navigation engine for autonomous nano-drones capable of closed-loop end-to-end DNN-based visual navigation. To achieve this goal we developed a complete methodology for parallel execution of complex DNNs directly on-bard of resource-constrained milliwatt-scale nodes. Our system is based on GAP8, a novel parallel ultra-low-power computing platform, and a 27 g commercial, open-source CrazyFlie 2.0 nano-quadrotor. As part of our general methodology we discuss the software mapping techniques that enable the state-of-the-art deep convolutional neural network presented in [1] to be fully executed on-board within a strict 6 fps real-time constraint with no compromise in terms of flight results, while all processing is done with only 64 mW on average. Our navigation engine is flexible and can be used to span a wide performance range: at its peak performance corner it achieves 18 fps while still consuming on average just 3.5% of the power envelope of the deployed nano-aircraft.

arXiv comment15 pages, 13 figures, 5 tables, 2 listings, accepted for publication in the IEEE Internet of Things Journal (IEEE IOTJ)

References in corpus (5)

• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Aggressive Quadrotor Flight through Narrow Gaps with Onboard Sensing and Computing using Active Vision
• Efficient Optical flow and Stereo Vision for Velocity Estimation and Obstacle Avoidance on an Autonomous Pocket Drone
• An IoT Endpoint System-on-Chip for Secure and Energy-Efficient Near-Sensor Analytics
• Local Histogram Matching for Efficient Optical Flow Computation Applied to Velocity Estimation on Pocket Drones

Cited by in corpus (30)

• A General Framework for Uncertainty Estimation in Deep Learning
• Machine Learning for Microcontroller-Class Hardware: A Review
• Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flight
• PULP-NN: Accelerating Quantized Neural Networks on Parallel Ultra-Low-Power RISC-V Processors
• DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• Sound Event Detection with Binary Neural Networks on Tightly Power-Constrained IoT Devices
• Improving Autonomous Nano-drones Performance via Automated End-to-End Optimization and Deployment of DNNs
• Learning to Seek: Autonomous Source Seeking with Deep Reinforcement Learning Onboard a Nano Drone Microcontroller
• HULK-V: a Heterogeneous Ultra-low-power Linux capable RISC-V SoC
• Additive Noise Annealing and Approximation Properties of Quantized Neural Networks
• Low Latency Visual Inertial Odometry with On-Sensor Accelerated Optical Flow for Resource-Constrained UAVs
• Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight
• VisualEchoes: Spatial Image Representation Learning through Echolocation
• Visual Odometry with Neuromorphic Resonator Networks
• FANN-on-MCU: An Open-Source Toolkit for Energy-Efficient Neural Network Inference at the Edge of the Internet of Things
• Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles
• A Fast and Accurate Optical Flow Camera for Resource-Constrained Edge Applications
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• Parallelizing Optical Flow Estimation on an Ultra-Low Power RISC-V Cluster for Nano-UAV Navigation
• Segregated FLS Processing Cores for V/STOL Autonomous Landing Guidance Assistant System using FPGA
• AutoPilot: Automating SoC Design Space Exploration for SWaP Constrained Autonomous UAVs
• Adaptive Prognostic Malfunction Based Processor for Autonomous Landing Guidance Assistance System Using FPGA
• Roofline Model for UAVs:A Bottleneck Analysis Tool for Designing Compute Systems for Autonomous Drones
• Learning Depth With Very Sparse Supervision
• Always-On 674uW @ 4GOP/s Error Resilient Binary Neural Networks with Aggressive SRAM Voltage Scaling on a 22nm IoT End-Node
• WiFi-Inertial Indoor Pose Estimation for Micro Aerial Vehicles
• A Flexible Exoskeleton for Collision Resilience
• XpulpNN: Enabling Energy Efficient and Flexible Inference of Quantized Neural Network on RISC-V based IoT End Nodes
• Training Lightweight CNNs for Human-Nanodrone Proximity Interaction from Small Datasets using Background Randomization