An Information-Theoretic Analysis of The Cost of Decentralization for Learning and Inference Under Privacy Constraints

Abstract

In vertical federated learning (FL), the features of a data sample are distributed across multiple agents. As such, inter-agent collaboration can be beneficial not only during the learning phase, as is the case for standard horizontal FL, but also during the inference phase. A fundamental theoretical question in this setting is how to quantify the cost, or performance loss, of decentralization for learning and/or inference. In this paper, we consider general supervised learning problems with any number of agents, and provide a novel information-theoretic quantification of the cost of decentralization in the presence of privacy constraints on inter-agent communication within a Bayesian framework. The cost of decentralization for learning and/or inference is shown to be quantified in terms of conditional mutual information terms involving features and label variables.

Under review

References in corpus (6)

• Feature Inference Attack on Model Predictions in Vertical Federated Learning
• Differential Privacy as a Mutual Information Constraint
• A Communication Efficient Collaborative Learning Framework for Distributed Features
• Privacy-Preserving Asynchronous Federated Learning Algorithms for Multi-Party Vertically Collaborative Learning
• Federated Learning for Coalition Operations
• Rate-Distortion Analysis of Minimum Excess Risk in Bayesian Learning