OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning

Abstract

Federated learning (FL) has emerged as a promising approach to collaboratively train machine learning models across multiple edge devices while preserving privacy. The success of FL hinges on the efficiency of participating models and their ability to handle the unique challenges of distributed learning. While several variants of Vision Transformer (ViT) have shown great potential as alternatives to modern convolutional neural networks (CNNs) for centralized training, the unprecedented size and higher computational demands hinder their deployment on resource-constrained edge devices, challenging their widespread application in FL. Since client devices in FL typically have limited computing resources and communication bandwidth, models intended for such devices must strike a balance between model size, computational efficiency, and the ability to adapt to the diverse and non-IID data distributions encountered in FL. To address these challenges, we propose OnDev-LCT: Lightweight Convolutional Transformers for On-Device vision tasks with limited training data and resources. Our models incorporate image-specific inductive biases through the LCT tokenizer by leveraging efficient depthwise separable convolutions in residual linear bottleneck blocks to extract local features, while the multi-head self-attention (MHSA) mechanism in the LCT encoder implicitly facilitates capturing global representations of images. Extensive experiments on benchmark image datasets indicate that our models outperform existing lightweight vision models while having fewer parameters and lower computational demands, making them suitable for FL scenarios with data heterogeneity and communication bottlenecks.

Published in Neural Networks

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Linformer: Self-Attention with Linear Complexity
• How I failed machine learning in medical imaging -- shortcomings and recommendations
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Escaping the Big Data Paradigm with Compact Transformers
• Separable Self-attention for Mobile Vision Transformers
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Towards Accurate Post-Training Quantization for Vision Transformer
• Vision Transformer Pruning
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• Mobile-Former: Bridging MobileNet and Transformer
• From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
• Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
• pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning
• Oscillation-free Quantization for Low-bit Vision Transformers
• Personalized Federated Learning with Hidden Information on Personalized Prior

Cited by in corpus (1)

• CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning