An FCNN-Based Super-Resolution mmWave Radar Framework for Contactless Musical Instrument Interface

Abstract

In this article, we propose a framework for contactless human-computer interaction (HCI) using novel tracking techniques based on deep learning-based super-resolution and tracking algorithms. Our system offers unprecedented high-resolution tracking of hand position and motion characteristics by leveraging spatial and temporal features embedded in the reflected radar waveform. Rather than classifying samples from a predefined set of hand gestures, as common in existing work on deep learning with mmWave radar, our proposed imager employs a regressive full convolutional neural network (FCNN) approach to improve localization accuracy by spatial super-resolution. While the proposed techniques are suitable for a host of tracking applications, this article focuses on their application as a musical interface to demonstrate the robustness of the gesture sensing pipeline and deep learning signal processing chain. The user can control the instrument by varying the position and velocity of their hand above the vertically-facing sensor. By employing a commercially available multiple-input-multiple-output (MIMO) radar rather than a traditional optical sensor, our framework demonstrates the efficacy of the mmWave sensing modality for fine motion tracking and offers an elegant solution to a host of HCI tasks. Additionally, we provide a freely available software package and user interface for controlling the device, streaming the data to MATLAB in real-time, and increasing accessibility to the signal processing and device interface functionality utilized in this article.

arXiv commentAccepted to IEEE Transactions on Multimedia

References in corpus (6)

• Fast and Robust Hand Tracking Using Detection-Guided Optimization
• Micro Hand Gesture Recognition System Using Ultrasonic Active Sensing
• High-level programming and control for industrial robotics: using a hand-held accelerometer-based input device for gesture and posture recognition
• Improved Static Hand Gesture Classification on Deep Convolutional Neural Networks using Novel Sterile Training Technique
• Near-Field MIMO-ISAR Millimeter-Wave Imaging
• Joint Range and Angle Estimation for FMCW MIMO Radar and Its Application

Cited by in corpus (4)

• Efficient 3-D Near-Field MIMO-SAR Imaging for Irregular Scanning Geometries
• Efficient CNN-based Super Resolution Algorithms for mmWave Mobile Radar Imaging
• Deep Learning-Based Multiband Signal Fusion for 3-D SAR Super-Resolution
• A Vision Transformer Approach for Efficient Near-Field Irregular SAR Super-Resolution