Federated Neural Architecture Search

Abstract

To preserve user privacy while enabling mobile intelligence, techniques have been proposed to train deep neural networks on decentralized data. However, training over decentralized data makes the design of neural architecture quite difficult as it already was. Such difficulty is further amplified when designing and deploying different neural architectures for heterogeneous mobile platforms. In this work, we propose an automatic neural architecture search into the decentralized training, as a new DNN training paradigm called Federated Neural Architecture Search, namely federated NAS. To deal with the primary challenge of limited on-client computational and communication resources, we present FedNAS, a highly optimized framework for efficient federated NAS. FedNAS fully exploits the key opportunity of insufficient model candidate re-training during the architecture search process, and incorporates three key optimizations: parallel candidates training on partial clients, early dropping candidates with inferior performance, and dynamic round numbers. Tested on large-scale datasets and typical CNN architectures, FedNAS achieves comparable model accuracy as state-of-the-art NAS algorithm that trains models with centralized data, and also reduces the client cost by up to two orders of magnitude compared to a straightforward design of federated NAS.

References in corpus (13)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Neural Architecture Search with Reinforcement Learning
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning for Mobile Keyboard Prediction
• Learning Differentially Private Recurrent Language Models
• Secure Federated Transfer Learning
• Efficient Neural Architecture Search via Parameter Sharing
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• LEAF: A Benchmark for Federated Settings
• Path-Level Network Transformation for Efficient Architecture Search
• Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search
• Not Just Privacy: Improving Performance of Private Deep Learning in Mobile Cloud

Cited by in corpus (6)

• FedML: A Research Library and Benchmark for Federated Machine Learning
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Accelerating Vertical Federated Learning
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Federated Learning on Non-IID Data: A Survey
• Direct Federated Neural Architecture Search