FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks

Abstract

Knowledge distillation (KD) has demonstrated its effectiveness to boost the performance of graph neural networks (GNNs), where its goal is to distill knowledge from a deeper teacher GNN into a shallower student GNN. However, it is actually difficult to train a satisfactory teacher GNN due to the well-known over-parametrized and over-smoothing issues, leading to invalid knowledge transfer in practical applications. In this paper, we propose the first Free-direction Knowledge Distillation framework via Reinforcement learning for GNNs, called FreeKD, which is no longer required to provide a deeper well-optimized teacher GNN. The core idea of our work is to collaboratively build two shallower GNNs in an effort to exchange knowledge between them via reinforcement learning in a hierarchical way. As we observe that one typical GNN model often has better and worse performances at different nodes during training, we devise a dynamic and free-direction knowledge transfer strategy that consists of two levels of actions: 1) node-level action determines the directions of knowledge transfer between the corresponding nodes of two networks; and then 2) structure-level action determines which of the local structures generated by the node-level actions to be propagated. In essence, our FreeKD is a general and principled framework which can be naturally compatible with GNNs of different architectures. Extensive experiments on five benchmark datasets demonstrate our FreeKD outperforms two base GNNs in a large margin, and shows its efficacy to various GNNs. More surprisingly, our FreeKD has comparable or even better performance than traditional KD algorithms that distill knowledge from a deeper and stronger teacher GNN.

Accepted to KDD 2022

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• A Brief Survey of Deep Reinforcement Learning
• Simplifying Graph Convolutional Networks
• Simple and Deep Graph Convolutional Networks
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Topology-Imbalance Learning for Semi-Supervised Node Classification
• RMM: Reinforced Memory Management for Class-Incremental Learning
• ROD: Reception-aware Online Distillation for Sparse Graphs

Cited by in corpus (1)

• MuGSI: Distilling GNNs with Multi-Granularity Structural Information for Graph Classification