Federated Learning Of Out-Of-Vocabulary Words

Abstract

We demonstrate that a character-level recurrent neural network is able to learn out-of-vocabulary (OOV) words under federated learning settings, for the purpose of expanding the vocabulary of a virtual keyboard for smartphones without exporting sensitive text to servers. High-frequency words can be sampled from the trained generative model by drawing from the joint posterior directly. We study the feasibility of the approach in two settings: (1) using simulated federated learning on a publicly available non-IID per-user dataset from a popular social networking website, (2) using federated learning on data hosted on user mobile devices. The model achieves good recall and precision compared to ground-truth OOV words in setting (1). With (2) we demonstrate the practicality of this approach by showing that we can learn meaningful OOV words with good character-level prediction accuracy and cross entropy loss.

References in corpus (2)

• Sequence to Sequence Learning with Neural Networks
• Practical Secure Aggregation for Federated Learning on User-Held Data

Cited by in corpus (20)

• Three Approaches for Personalization with Applications to Federated Learning
• A Field Guide to Federated Optimization
• Edge Intelligence: Architectures, Challenges, and Applications
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• Communication Efficiency in Federated Learning: Achievements and Challenges
• FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
• Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
• Hierarchical Federated Learning through LAN-WAN Orchestration
• Secure Federated Submodel Learning
• A Systematic Literature Review on Federated Learning: From A Model Quality Perspective
• Dynamic Fusion based Federated Learning for COVID-19 Detection
• Blockchain-based Federated Learning for Device Failure Detection in Industrial IoT
• Self-supervised Cross-silo Federated Neural Architecture Search
• Privacy-preserving Decentralized Aggregation for Federated Learning
• Efficient Privacy Preserving Edge Computing Framework for Image Classification
• LIA: Privacy-Preserving Data Quality Evaluation in Federated Learning Using a Lazy Influence Approximation
• Preliminary Steps Towards Federated Sentiment Classification
• Reliability and Performance Assessment of Federated Learning on Clinical Benchmark Data
• Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments