Towards Practical Few-shot Federated NLP

Abstract

Transformer-based pre-trained models have emerged as the predominant solution for natural language processing (NLP). Fine-tuning such pre-trained models for downstream tasks often requires a considerable amount of labeled private data. In practice, private data is often distributed across heterogeneous mobile devices and may be prohibited from being uploaded. Moreover, well-curated labeled data is often scarce, presenting an additional challenge. To address these challenges, we first introduce a data generator for federated few-shot learning tasks, which encompasses the quantity and skewness of scarce labeled data in a realistic setting. Subsequently, we propose AUG-FedPrompt, a prompt-based federated learning system that exploits abundant unlabeled data for data augmentation. Our experiments indicate that AUG-FedPrompt can perform on par with full-set fine-tuning with a limited amount of labeled data. However, such competitive performance comes at a significant system cost.

arXiv commentEuroSys23 workshop

References in corpus (13)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Character-level Convolutional Networks for Text Classification
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Training Deep Nets with Sublinear Memory Cost
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation
• How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Unsupervised Prompt Learning for Vision-Language Models
• BERT Loses Patience: Fast and Robust Inference with Early Exit
• Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
• Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspective

Cited by in corpus (1)

• Federated Few-Shot Learning for Mobile NLP