Federated Learning for Mobile Keyboard Prediction

Abstract

We train a recurrent neural network language model using a distributed, on-device learning framework called federated learning for the purpose of next-word prediction in a virtual keyboard for smartphones. Server-based training using stochastic gradient descent is compared with training on client devices using the Federated Averaging algorithm. The federated algorithm, which enables training on a higher-quality dataset for this use case, is shown to achieve better prediction recall. This work demonstrates the feasibility and benefit of training language models on client devices without exporting sensitive user data to servers. The federated learning environment gives users greater control over the use of their data and simplifies the task of incorporating privacy by default with distributed training and aggregation across a population of client devices.

arXiv comment7 pages, 4 figures

References in corpus (9)

• LSTM: A Search Space Odyssey
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Character-Aware Neural Language Models
• Exploring the Limits of Language Modeling
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Practical Secure Aggregation for Federated Learning on User-Held Data
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Neural Networks for Text Correction and Completion in Keyboard Decoding
• Mobile Keyboard Input Decoding with Finite-State Transducers

Cited by in corpus (217)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Towards Federated Learning at Scale: System Design
• How To Backdoor Federated Learning
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Sustainable AI: Environmental Implications, Challenges and Opportunities
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Adaptive Personalized Federated Learning
• Three Approaches for Personalization with Applications to Federated Learning
• FedGNN: Federated Graph Neural Network for Privacy-Preserving Recommendation
• Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems
• Agnostic Federated Learning
• Electrical Load Forecasting Using Edge Computing and Federated Learning
• Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System
• FMore: An Incentive Scheme of Multi-dimensional Auction for Federated Learning in MEC
• Personalized Federated Learning with Moreau Envelopes
• Federated Learning for Healthcare Domain - Pipeline, Applications and Challenges
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• Federated Learning: A Signal Processing Perspective
• A Field Guide to Federated Optimization
• Hierarchical Federated Learning with Quantization: Convergence Analysis and System Design
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review
• Edge Intelligence: Architectures, Challenges, and Applications
• FedParking: A Federated Learning based Parking Space Estimation with Parked Vehicle assisted Edge Computing
• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• FedDPGAN: Federated Differentially Private Generative Adversarial Networks Framework for the Detection of COVID-19 Pneumonia
• Flower: A Friendly Federated Learning Research Framework
• Federated Learning Of Out-Of-Vocabulary Words
• FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks
• Preserving Privacy and Security in Federated Learning
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
• Federated Learning with Spiking Neural Networks
• A Critical Review on the Use (and Misuse) of Differential Privacy in Machine Learning
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Concept drift detection and adaptation for federated and continual learning
• Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework
• LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets
• Free-riders in Federated Learning: Attacks and Defenses
• Salvaging Federated Learning by Local Adaptation
• Real-World Image Datasets for Federated Learning
• Federated Learning for Healthcare Informatics
• FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems
• First Analysis of Local GD on Heterogeneous Data
• A Systematic Survey of Blockchained Federated Learning
• FetchSGD: Communication-Efficient Federated Learning with Sketching
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• Federated Learning Under Intermittent Client Availability and Time-Varying Communication Constraints
• AdaCliP: Adaptive Clipping for Private SGD
• A Comprehensive Survey of Incentive Mechanism for Federated Learning
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• Communication Efficiency in Federated Learning: Achievements and Challenges
• Data Poisoning Attacks Against Federated Learning Systems
• Enhanced Security and Privacy via Fragmented Federated Learning
• Towards Energy Efficient Distributed Federated Learning for 6G Networks
• Oort: Efficient Federated Learning via Guided Participant Selection
• Generative Models for Effective ML on Private, Decentralized Datasets
• Triggerflow: Trigger-based Orchestration of Serverless Workflows
• Federated Learning Meets Natural Language Processing: A Survey
• FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximation
• Federated Transfer-Ordered-Personalized Learning for Driver Monitoring Application
• Tighter Theory for Local SGD on Identical and Heterogeneous Data
• Triggerflow: Trigger-based Orchestration of Serverless Workflows
• Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms
• Federated Learning with Buffered Asynchronous Aggregation
• The Skellam Mechanism for Differentially Private Federated Learning
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Faster On-Device Training Using New Federated Momentum Algorithm
• Eavesdrop the Composition Proportion of Training Labels in Federated Learning
• Fed-Focal Loss for imbalanced data classification in Federated Learning
• Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies
• Incentive Design for Efficient Federated Learning in Mobile Networks: A Contract Theory Approach
• Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
• Multi-Model Federated Learning
• Hierarchical Federated Learning through LAN-WAN Orchestration
• Papaya: Practical, Private, and Scalable Federated Learning
• FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers
• HashVFL: Defending Against Data Reconstruction Attacks in Vertical Federated Learning
• A Survey on Federated Learning and its Applications for Accelerating Industrial Internet of Things
• Secure Federated Submodel Learning
• Client-Edge-Cloud Hierarchical Federated Learning
• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• A Federated Learning Approach for Mobile Packet Classification
• Federated Reconstruction: Partially Local Federated Learning
• Federated Neural Architecture Search
• Enhancing the Privacy of Federated Learning with Sketching
• A Systematic Literature Review on Federated Learning: From A Model Quality Perspective
• The Tradeoff Between Privacy and Accuracy in Anomaly Detection Using Federated XGBoost
• Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy
• Local Adaptivity in Federated Learning: Convergence and Consistency
• On the Outsized Importance of Learning Rates in Local Update Methods
• FedJAX: Federated learning simulation with JAX
• Understanding Unintended Memorization in Federated Learning
• Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training
• Asynchronous Online Federated Learning for Edge Devices with Non-IID Data
• Federated Learning of N-gram Language Models
• Stragglers Are Not Disaster: A Hybrid Federated Learning Algorithm with Delayed Gradients
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
• Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing
• Self-supervised Cross-silo Federated Neural Architecture Search
• Byzantines can also Learn from History: Fall of Centered Clipping in Federated Learning
• A Principled Approach to Data Valuation for Federated Learning
• Training Production Language Models without Memorizing User Data
• Securing Federated Sensitive Topic Classification against Poisoning Attacks
• Incentives for Federated Learning: a Hypothesis Elicitation Approach
• Specialized federated learning using a mixture of experts
• Fast-adapting and Privacy-preserving Federated Recommender System
• A collaborative ensemble construction method for federated random forest
• FedCon: A Contrastive Framework for Federated Semi-Supervised Learning
• Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning
• Federated Learning for Distributed Energy-Efficient Resource Allocation
• To Talk or to Work: Flexible Communication Compression for Energy Efficient Federated Learning over Heterogeneous Mobile Edge Devices
• Layer-wise Characterization of Latent Information Leakage in Federated Learning
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• Network Support for High-performance Distributed Machine Learning
• EdgeRec: Recommender System on Edge in Mobile Taobao
• FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training
• Turn Signal Prediction: A Federated Learning Case Study
• Strategic Coalition for Data Pricing in IoT Data Markets
• Federated Composite Optimization
• FedSiam: Towards Adaptive Federated Semi-Supervised Learning
• Federated Self-Supervised Learning in Heterogeneous Settings: Limits of a Baseline Approach on HAR
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• Writing Across the World's Languages: Deep Internationalization for Gboard, the Google Keyboard
• Improving Accuracy of Federated Learning in Non-IID Settings
• Differentially Private Federated Learning with Laplacian Smoothing
• DistFL: Distribution-aware Federated Learning for Mobile Scenarios
• Stochastic Channel-Based Federated Learning for Medical Data Privacy Preserving
• Fed-EC: Bandwidth-Efficient Clustering-Based Federated Learning For Autonomous Visual Robot Navigation
• Multi-modal AsynDGAN: Learn From Distributed Medical Image Data without Sharing Private Information
• Towards On-Device Federated Learning: A Direct Acyclic Graph-based Blockchain Approach
• SAPAG: A Self-Adaptive Privacy Attack From Gradients
• Secure Byzantine-Robust Distributed Learning via Clustering
• Privacy-preserving Decentralized Aggregation for Federated Learning
• Improving Response Time of Home IoT Services in Federated Learning
• A Robust Federated Learning Approach for Combating Attacks Against IoT Systems Under non-IID Challenges
• Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning
• Real-time End-to-End Federated Learning: An Automotive Case Study
• Federated Learning in ASR: Not as Easy as You Think
• Efficient Privacy Preserving Edge Computing Framework for Image Classification
• ESMFL: Efficient and Secure Models for Federated Learning
• TextHide: Tackling Data Privacy in Language Understanding Tasks
• Heterogeneity-aware Cross-school Electives Recommendation: a Hybrid Federated Approach
• Voting-based Approaches For Differentially Private Federated Learning
• Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
• A Framework for testing Federated Learning algorithms using an edge-like environment
• Failure Prediction in Production Line Based on Federated Learning: An Empirical Study
• FederatedNILM: A Distributed and Privacy-preserving Framework for Non-intrusive Load Monitoring based on Federated Deep Learning
• Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shift
• Federated Learning of User Authentication Models
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• Addressing Class Imbalance in Federated Learning
• Privacy Preserving Stochastic Channel-Based Federated Learning with Neural Network Pruning
• Policy-Based Federated Learning
• PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization
• An Exploratory Analysis on Users' Contributions in Federated Learning
• Faithful Edge Federated Learning: Scalability and Privacy
• Robust Federated Learning with Attack-Adaptive Aggregation
• Incremental Layer-wise Self-Supervised Learning for Efficient Speech Domain Adaptation On Device
• LIA: Privacy-Preserving Data Quality Evaluation in Federated Learning Using a Lazy Influence Approximation
• Federated Imitation Learning: A Privacy Considered Imitation Learning Framework for Cloud Robotic Systems with Heterogeneous Sensor Data
• Federated learning with differential privacy and an untrusted aggregator
• Prophet: Proactive Candidate-Selection for Federated Learning by Predicting the Qualities of Training and Reporting Phases
• WAFFLE: Watermarking in Federated Learning
• Model Reuse with Reduced Kernel Mean Embedding Specification
• Federated Generative Adversarial Learning
• Federated Estimation of Causal Effects from Observational Data
• Federated Natural Language Generation for Personalized Dialogue System
• FedSup: A Communication-Efficient Federated Learning Fatigue Driving Behaviors Supervision Framework
• Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning
• Federated Acoustic Modeling For Automatic Speech Recognition
• Communication-Efficient Agnostic Federated Averaging
• Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
• Reward-Based 1-bit Compressed Federated Distillation on Blockchain
• Reliability and Performance Assessment of Federated Learning on Clinical Benchmark Data
• Adaptive Distillation for Decentralized Learning from Heterogeneous Clients
• TiFL: A Tier-based Federated Learning System
• Local SGD Optimizes Overparameterized Neural Networks in Polynomial Time
• Learn distributed GAN with Temporary Discriminators
• Covert Model Poisoning Against Federated Learning: Algorithm Design and Optimization
• A Communication-efficient Local Differentially Private Algorithm in Federated Optimization
• Adversarial Robustness through Bias Variance Decomposition: A New Perspective for Federated Learning
• FL-APU: A Software Architecture to Ease Practical Implementation of Cross-Silo Federated Learning
• Federated Learning With Highly Imbalanced Audio Data
• EasyFL: A Low-code Federated Learning Platform For Dummies
• Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks
• FedProf: Selective Federated Learning with Representation Profiling
• Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates
• Security and Privacy Preserving Deep Learning
• Marketplace for AI Models
• Improving the Algorithm of Deep Learning with Differential Privacy
• Data-Free Evaluation of User Contributions in Federated Learning
• Simeon -- Secure Federated Machine Learning Through Iterative Filtering
• Demystifying the Effects of Non-Independence in Federated Learning
• Communication Efficient Federated Learning with Adaptive Quantization
• Assessing the Feasibility of Web-Request Prediction Models on Mobile Platforms
• Strong Privacy and Utility Guarantee: Over-the-Air Statistical Estimation
• Byzantine-Resilient Federated Machine Learning via Over-the-Air Computation
• CatFedAvg: Optimising Communication-efficiency and Classification Accuracy in Federated Learning
• Loosely Coupled Federated Learning Over Generative Models
• Distributed Online Learning with Multiple Kernels
• Federated Classification using Parsimonious Functions in Reproducing Kernel Hilbert Spaces
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks
• Multiple Kernel-Based Online Federated Learning
• Management of Resource at the Network Edge for Federated Learning
• UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approach
• Secure Distributed Training at Scale
• An Experiment Study on Federated LearningTestbed
• Distantly Supervised Relation Extraction in Federated Settings
• Multiple Classification with Split Learning
• Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments
• A Personalized Federated Learning Algorithm: an Application in Anomaly Detection
• AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning