SCoTTi: Save Computation at Training Time with an adaptive framework

Abstract

On-device training is an emerging approach in machine learning where models are trained on edge devices, aiming to enhance privacy protection and real-time performance. However, edge devices typically possess restricted computational power and resources, making it challenging to perform computationally intensive model training tasks. Consequently, reducing resource consumption during training has become a pressing concern in this field. To this end, we propose SCoTTi (Save Computation at Training Time), an adaptive framework that addresses the aforementioned challenge. It leverages an optimizable threshold parameter to effectively reduce the number of neuron updates during training which corresponds to a decrease in memory and computation footprint. Our proposed approach demonstrates superior performance compared to the state-of-the-art methods regarding computational resource savings on various commonly employed benchmarks and popular architectures, including ResNets, MobileNet, and Swin-T.

References in corpus (8)

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Straggler-aware Distributed Learning: Communication Computation Latency Trade-off
• A Federated Deep Learning Framework for Privacy Preservation and Communication Efficiency
• Gradient Descent: The Ultimate Optimizer
• Optimized preprocessing and Tiny ML for Attention State Classification
• Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima
• Weight Pruning via Adaptive Sparsity Loss