Canoe : A System for Collaborative Learning for Neural Nets

Abstract

For highly distributed environments such as edge computing, collaborative learning approaches eschew the dependence on a global, shared model, in favor of models tailored for each location. Creating tailored models for individual learning contexts reduces the amount of data transfer, while collaboration among peers provides acceptable model performance. Collaboration assumes, however, the availability of knowledge transfer mechanisms, which are not trivial for deep learning models where knowledge isn't easily attributed to precise model slices. We present Canoe - a framework that facilitates knowledge transfer for neural networks. Canoe provides new system support for dynamically extracting significant parameters from a helper node's neural network and uses this with a multi-model boosting-based approach to improve the predictive performance of the target node. The evaluation of Canoe with different PyTorch and TensorFlow neural network models demonstrates that the knowledge transfer mechanism improves the model's adaptiveness to changes up to 3.5X compared to learning in isolation, while affording several magnitudes reduction in data movement costs compared to federated learning.

References in corpus (23)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• How transferable are features in deep neural networks?
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• Federated Learning: Strategies for Improving Communication Efficiency
• Recurrent Neural Network Regularization
• In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning
• Towards Federated Learning at Scale: System Design
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• Federated Learning with Personalization Layers
• EMNIST: an extension of MNIST to handwritten letters
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Large scale distributed neural network training through online distillation
• Revisiting the Importance of Individual Units in CNNs via Ablation
• Oort: Efficient Federated Learning via Guided Participant Selection
• Understanding trained CNNs by indexing neuron selectivity
• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies
• CodedPrivateML: A Fast and Privacy-Preserving Framework for Distributed Machine Learning
• Identifying and Controlling Important Neurons in Neural Machine Translation
• Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control
• Distilling Critical Paths in Convolutional Neural Networks
• Task-Adaptive Incremental Learning for Intelligent Edge Devices