Pool of Experts: Realtime Querying Specialized Knowledge in Massive Neural Networks

Abstract

In spite of the great success of deep learning technologies, training and delivery of a practically serviceable model is still a highly time-consuming process. Furthermore, a resulting model is usually too generic and heavyweight, and hence essentially goes through another expensive model compression phase to fit in a resource-limited device like embedded systems. Inspired by the fact that a machine learning task specifically requested by mobile users is often much simpler than it is supported by a massive generic model, this paper proposes a framework, called Pool of Experts (PoE), that instantly builds a lightweight and task-specific model without any training process. For a realtime model querying service, PoE first extracts a pool of primitive components, called experts, from a well-trained and sufficiently generic network by exploiting a novel conditional knowledge distillation method, and then performs our train-free knowledge consolidation to quickly combine necessary experts into a lightweight network for a target task. Thanks to this train-free property, in our thorough empirical study, PoE can build a fairly accurate yet compact model in a realtime manner, whereas it takes a few minutes per query for the other training methods to achieve a similar level of the accuracy.

In SIGMOD/PODS 2021

References in corpus (5)

• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Compressing Neural Networks with the Hashing Trick
• Focus: Querying Large Video Datasets with Low Latency and Low Cost