Federated Few-Shot Learning for Mobile NLP

Abstract

Natural language processing (NLP) sees rich mobile applications. To support various language understanding tasks, a foundation NLP model is often fine-tuned in a federated, privacy-preserving setting (FL). This process currently relies on at least hundreds of thousands of labeled training samples from mobile clients; yet mobile users often lack willingness or knowledge to label their data. Such an inadequacy of data labels is known as a few-shot scenario; it becomes the key blocker for mobile NLP applications. For the first time, this work investigates federated NLP in the few-shot scenario (FedFSL). By retrofitting algorithmic advances of pseudo labeling and prompt learning, we first establish a training pipeline that delivers competitive accuracy when only 0.05% (fewer than 100) of the training data is labeled and the remaining is unlabeled. To instantiate the workflow, we further present a system FeS, addressing the high execution cost with novel designs. (1) Curriculum pacing, which injects pseudo labels to the training workflow at a rate commensurate to the learning progress; (2) Representational diversity, a mechanism for selecting the most learnable data, only for which pseudo labels will be generated; (3) Co-planning of a model's training depth and layer capacity. Together, these designs reduce the training delay, client energy, and network traffic by up to 46.0$\times$, 41.2$\times$ and 3000.0$\times$, respectively. Through algorithm/system co-design, FFNLP demonstrates that FL can apply to challenging settings where most training samples are unlabeled.

arXiv commentMobiCom 2023

References in corpus (37)

• Prototypical Networks for Few-shot Learning
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Character-level Convolutional Networks for Text Classification
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• On the Opportunities and Risks of Foundation Models
• Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Towards Federated Learning at Scale: System Design
• Generalizing from a Few Examples: A Survey on Few-Shot Learning
• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Towards a Unified View of Parameter-Efficient Transfer Learning
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• Gradient Sparsification for Communication-Efficient Distributed Optimization
• One-Shot Federated Learning
• Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• PPT: Pre-trained Prompt Tuning for Few-shot Learning
• How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations
• Selective Annotation Makes Language Models Better Few-Shot Learners
• Unsupervised Prompt Learning for Vision-Language Models
• Oort: Efficient Federated Learning via Guided Participant Selection
• FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Semi-supervised Federated Learning for Activity Recognition
• Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labeling
• LAMP: Extracting Text from Gradients with Language Model Priors
• FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clients
• From Cloud to Edge: A First Look at Public Edge Platforms
• Towards Practical Few-shot Federated NLP
• FedAdapter: Efficient Federated Learning for Modern NLP
• A Secure and Efficient Federated Learning Framework for NLP
• Mandheling: Mixed-Precision On-Device DNN Training with DSP Offloading
• Unfreeze with Care: Space-Efficient Fine-Tuning of Semantic Parsing Models

Cited by in corpus (1)

• Uncovering Gradient Inversion Risks in Practical Language Model Training