Topological Persistence Guided Knowledge Distillation for Wearable Sensor Data

Abstract

Deep learning methods have achieved a lot of success in various applications involving converting wearable sensor data to actionable health insights. A common application areas is activity recognition, where deep-learning methods still suffer from limitations such as sensitivity to signal quality, sensor characteristic variations, and variability between subjects. To mitigate these issues, robust features obtained by topological data analysis (TDA) have been suggested as a potential solution. However, there are two significant obstacles to using topological features in deep learning: (1) large computational load to extract topological features using TDA, and (2) different signal representations obtained from deep learning and TDA which makes fusion difficult. In this paper, to enable integration of the strengths of topological methods in deep-learning for time-series data, we propose to use two teacher networks, one trained on the raw time-series data, and another trained on persistence images generated by TDA methods. The distilled student model utilizes only the raw time-series data at test-time. This approach addresses both issues. The use of KD with multiple teachers utilizes complementary information, and results in a compact model with strong supervisory features and an integrated richer representation. To assimilate desirable information from different modalities, we design new constraints, including orthogonality imposed on feature correlation maps for improving feature expressiveness and allowing the student to easily learn from the teacher. Also, we apply an annealing strategy in KD for fast saturation and better accommodation from different features, while the knowledge gap between the teachers and student is reduced. Finally, a robust student model is distilled, which uses only the time-series data as an input, while implicitly preserving topological features.

Engineering Applications of Artificial Intelligence 130, 107719

References in corpus (16)

• Knowledge Distillation: A Survey
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Deep Learning for Sensor-based Activity Recognition: A Survey
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Time Series Data Augmentation for Deep Learning: A Survey
• Algorithms for Learning Kernels Based on Centered Alignment
• Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables
• Video-SwinUNet: Spatio-temporal Deep Learning Framework for VFSS Instance Segmentation
• Adaptive Multi-Teacher Multi-level Knowledge Distillation
• Knowledge Distillation from A Stronger Teacher
• A Short Survey of Topological Data Analysis in Time Series and Systems Analysis
• Role of Data Augmentation Strategies in Knowledge Distillation for Wearable Sensor Data
• Topological Attention for Time Series Forecasting
• Role of Orthogonality Constraints in Improving Properties of Deep Networks for Image Classification
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective