Topology-aware Embedding Memory for Continual Learning on Expanding Networks

Abstract

Memory replay based techniques have shown great success for continual learning with incrementally accumulated Euclidean data. Directly applying them to continually expanding networks, however, leads to the potential memory explosion problem due to the need to buffer representative nodes and their associated topological neighborhood structures. To this end, we systematically analyze the key challenges in the memory explosion problem, and present a general framework, \textit{i.e.}, Parameter Decoupled Graph Neural Networks (PDGNNs) with Topology-aware Embedding Memory (TEM), to tackle this issue. The proposed framework not only reduces the memory space complexity from $\mathcal{O}(n{d}^L)$ to $\mathcal{O}(n)$~\footnote{$n$: memory budget, $d$: average node degree, $L$: the radius of the GNN receptive field}, but also fully utilizes the topological information for memory replay. Specifically, PDGNNs decouple trainable parameters from the computation ego-subnetwork via \textit{Topology-aware Embeddings} (TEs), which compress ego-subnetworks into compact vectors (\textit{i.e.}, TEs) to reduce the memory consumption. Based on this framework, we discover a unique \textit{pseudo-training effect} in continual learning on expanding networks and this effect motivates us to develop a novel \textit{coverage maximization sampling} strategy that can enhance the performance with a tight memory budget. Thorough empirical studies demonstrate that, by tackling the memory explosion problem and incorporating topological information into memory replay, PDGNNs with TEM significantly outperform state-of-the-art techniques, especially in the challenging class-incremental setting.

This paper has been accepted by KDD 2024

References in corpus (13)

• Overcoming catastrophic forgetting in neural networks
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs
• Few-Shot Graph Learning for Molecular Property Prediction
• Graph Attention Multi-Layer Perceptron
• Streaming Graph Neural Networks via Continual Learning
• GraphSAIL: Graph Structure Aware Incremental Learning for Recommender Systems
• Lifelong Learning of Graph Neural Networks for Open-World Node Classification
• Lifelong Learning on Evolving Graphs Under the Constraints of Imbalanced Classes and New Classes
• Graph Edge Convolutional Neural Networks for Skeleton Based Action Recognition
• Open-World Lifelong Graph Learning