FeatureNorm: L2 Feature Normalization for Dynamic Graph Embedding

Abstract

Dynamic graphs arise in a plethora of practical scenarios such as social networks, communication networks, and financial transaction networks. Given a dynamic graph, it is fundamental and essential to learn a graph representation that is expected not only to preserve structural proximity but also jointly capture the time-evolving patterns. Recently, graph convolutional network (GCN) has been widely explored and used in non-Euclidean application domains. The main success of GCN, especially in handling dependencies and passing messages within nodes, lies in its approximation to Laplacian smoothing. As a matter of fact, this smoothing technique can not only encourage must-link node pairs to get closer but also push cannot-link pairs to shrink together, which potentially cause serious feature shrink or oversmoothing problem, especially when stacking graph convolution in multiple layers or steps. For learning time-evolving patterns, a natural solution is to preserve historical state and combine it with the current interactions to obtain the most recent representation. Then the serious feature shrink or oversmoothing problem could happen when stacking graph convolution explicitly or implicitly according to current prevalent methods, which would make nodes too similar to distinguish each other. To solve this problem in dynamic graph embedding, we analyze the shrinking properties in the node embedding space at first, and then design a simple yet versatile method, which exploits L2 feature normalization constraint to rescale all nodes to hypersphere of a unit ball so that nodes would not shrink together, and yet similar nodes can still get closer. Extensive experiments on four real-world dynamic graph datasets compared with competitive baseline models demonstrate the effectiveness of the proposed method.

arXiv commentICDM 2020;

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• DeepWalk: Online Learning of Social Representations
• LINE: Large-scale Information Network Embedding
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• T-GCN: A Temporal Graph ConvolutionalNetwork for Traffic Prediction
• Representation Learning on Graphs with Jumping Knowledge Networks
• Variational Graph Auto-Encoders
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• dyngraph2vec: Capturing Network Dynamics using Dynamic Graph Representation Learning
• Gated Graph Sequence Neural Networks
• Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning
• Attributed Network Embedding for Learning in a Dynamic Environment
• DynGEM: Deep Embedding Method for Dynamic Graphs
• Scalable Link Prediction in Dynamic Networks via Non-Negative Matrix Factorization
• GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
• PairNorm: Tackling Oversmoothing in GNNs
• Node Embedding over Temporal Graphs
• Unifying Graph Convolutional Neural Networks and Label Propagation
• Dynamic Graph Representation Learning via Self-Attention Networks
• Spatio-Temporal Neural Networks for Space-Time Series Forecasting and Relations Discovery
• Inductive Representation Learning on Temporal Graphs