Canoe : A System for Collaborative Learning for Neural Nets

Abstract

For highly distributed environments such as edge computing, collaborative learning approaches eschew the dependence on a global, shared model, in favor of models tailored for each location. Creating tailored models for individual learning contexts reduces the amount of data transfer, while collaboration among peers provides acceptable model performance. Collaboration assumes, however, the availability of knowledge transfer mechanisms, which are not trivial for deep learning models where knowledge isn't easily attributed to precise model slices. We present Canoe - a framework that facilitates knowledge transfer for neural networks. Canoe provides new system support for dynamically extracting significant parameters from a helper node's neural network and uses this with a multi-model boosting-based approach to improve the predictive performance of the target node. The evaluation of Canoe with different PyTorch and TensorFlow neural network models demonstrates that the knowledge transfer mechanism improves the model's adaptiveness to changes up to 3.5X compared to learning in isolation, while affording several magnitudes reduction in data movement costs compared to federated learning.

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• How transferable are features in deep neural networks?
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• Recurrent Neural Network Regularization
• Towards Federated Learning at Scale: System Design
• Federated Learning with Personalization Layers
• EMNIST: an extension of MNIST to handwritten letters
• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies
• Task-Adaptive Incremental Learning for Intelligent Edge Devices