Self-supervised Cross-silo Federated Neural Architecture Search

Abstract

Federated Learning (FL) provides both model performance and data privacy for machine learning tasks where samples or features are distributed among different parties. In the training process of FL, no party has a global view of data distributions or model architectures of other parties. Thus the manually-designed architectures may not be optimal. In the past, Neural Architecture Search (NAS) has been applied to FL to address this critical issue. However, existing Federated NAS approaches require prohibitive communication and computation effort, as well as the availability of high-quality labels. In this work, we present Self-supervised Vertical Federated Neural Architecture Search (SS-VFNAS) for automating FL where participants hold feature-partitioned data, a common cross-silo scenario called Vertical Federated Learning (VFL). In the proposed framework, each party first conducts NAS using self-supervised approach to find a local optimal architecture with its own data. Then, parties collaboratively improve the local optimal architecture in a VFL framework with supervision. We demonstrate experimentally that our approach has superior performance, communication efficiency and privacy compared to Federated NAS and is capable of generating high-performance and highly-transferable heterogeneous architectures even with insufficient overlapping samples, providing automation for those parties without deep learning expertise.

arXiv commentThis work has been submitted to the IEEE for possible publication

References in corpus (28)

• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Improved Baselines with Momentum Contrastive Learning
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• Federated Learning for Mobile Keyboard Prediction
• Secure Federated Transfer Learning
• Federated Machine Learning: Concept and Applications
• A generic framework for privacy preserving deep learning
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Protection Against Reconstruction and Its Applications in Private Federated Learning
• Split learning for health: Distributed deep learning without sharing raw patient data
• Federated Learning for Emoji Prediction in a Mobile Keyboard
• Federated Learning Of Out-Of-Vocabulary Words
• A Framework for Evaluating Gradient Leakage Attacks in Federated Learning
• Parallel Distributed Logistic Regression for Vertical Federated Learning without Third-Party Coordinator
• Split Learning for collaborative deep learning in healthcare
• Multi-Participant Multi-Class Vertical Federated Learning
• FetchSGD: Communication-Efficient Federated Learning with Sketching
• Mitigating Backdoor Attacks in Federated Learning
• A Communication Efficient Collaborative Learning Framework for Distributed Features
• Learning Privately over Distributed Features: An ADMM Sharing Approach
• Asymmetrical Vertical Federated Learning
• Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search
• Real-time Federated Evolutionary Neural Architecture Search
• Self-supervised Neural Architecture Search

Cited by in corpus (3)

• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• Federated Learning on Non-IID Data: A Survey