Learning Private Neural Language Modeling with Attentive Aggregation

Abstract

Mobile keyboard suggestion is typically regarded as a word-level language modeling problem. Centralized machine learning technique requires massive user data collected to train on, which may impose privacy concerns for sensitive personal typing data of users. Federated learning (FL) provides a promising approach to learning private language modeling for intelligent personalized keyboard suggestion by training models in distributed clients rather than training in a central server. To obtain a global model for prediction, existing FL algorithms simply average the client models and ignore the importance of each client during model aggregation. Furthermore, there is no optimization for learning a well-generalized global model on the central server. To solve these problems, we propose a novel model aggregation with the attention mechanism considering the contribution of clients models to the global model, together with an optimization technique during server aggregation. Our proposed attentive aggregation method minimizes the weighted distance between the server model and client models through iterative parameters updating while attends the distance between the server model and client models. Through experiments on two popular language modeling datasets and a social media dataset, our proposed method outperforms its counterparts in terms of perplexity and communication cost in most settings of comparison.

arXiv comment8 pages, 5 figures, to appear in IJCNN 2019

References in corpus (2)

• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Tensor Factorization for Computational Phenotyping

Cited by in corpus (11)

• FedGNN: Federated Graph Neural Network for Privacy-Preserving Recommendation
• Dual Attention-Based Federated Learning for Wireless Traffic Prediction
• Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning
• Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning
• Secure and Efficient Federated Learning Through Layering and Sharding Blockchain
• Gradient Compression and Correlation Driven Federated Learning for Wireless Traffic Prediction
• Cellular Traffic Prediction via Byzantine-robust Asynchronous Federated Learning
• Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring
• Transformers with Attentive Federated Aggregation for Time Series Stock Forecasting
• Gradual Federated Learning with Simulated Annealing
• DUA-D2C: Dynamic Uncertainty Aware Method for Overfitting Remediation in Deep Learning