Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO

Abstract

Smart glasses are rapidly gaining advanced functions thanks to cutting-edge computing technologies, especially accelerated hardware architectures, and tiny Artificial Intelligence (AI) algorithms. However, integrating AI into smart glasses featuring a small form factor and limited battery capacity remains challenging for a satisfactory user experience. To this end, this paper proposes the design of a smart glasses platform for always-on on-device object detection with an all-day battery lifetime. The proposed platform is based on GAP9, a novel multi-core RISC-V processor from Greenwaves Technologies. Additionally, a family of sub-million parameter TinyissimoYOLO networks are proposed. They are benchmarked on established datasets, capable of differentiating up to 80 classes on MS-COCO. Evaluations on the smart glasses prototype demonstrate TinyissimoYOLO's inference latency of only 17ms and consuming 1.59mJ energy per inference. An end-to-end latency of 56ms is achieved which is equivalent to 18 frames per seconds (FPS) with a total power consumption of 62.9mW. This ensures continuous system runtime of up to 9.3 hours on a 154mAh battery. These results outperform MCUNet (TinyNAS+TinyEngine), which runs a simpler task (image classification) at just 7.3 FPS, while the 18 FPS achieved in this paper even include image-capturing, network inference, and detection post-processing. The algorithm's code is released open with this paper and can be found here: https://github.com/ETH-PBL/TinyissimoYOLO

This paper has been accepted for publication at ECCV 2024 Workshops, Milan, 2024

References in corpus (15)

• YOLOv4: Optimal Speed and Accuracy of Object Detection
• YOLOX: Exceeding YOLO Series in 2021
• YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications
• A Survey of Large Language Models
• YOLOv10: Real-Time End-to-End Object Detection
• BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View
• Machine Learning for Microcontroller-Class Hardware: A Review
• Survey of Machine Learning Accelerators
• PP-PicoDet: A Better Real-Time Object Detector on Mobile Devices
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• TinyissimoYOLO: A Quantized, Low-Memory Footprint, TinyML Object Detection Network for Low Power Microcontrollers
• Marsellus: A Heterogeneous RISC-V AI-IoT End-Node SoC with 2-to-8b DNN Acceleration and 30%-Boost Adaptive Body Biasing
• ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers
• Flexible and Fully Quantized Ultra-Lightweight TinyissimoYOLO for Ultra-Low-Power Edge Systems
• Exploring Automatic Gym Workouts Recognition Locally On Wearable Resource-Constrained Devices