189 citations · 1.7k across the 110 of their papers we have counts for
6 papers · 2 filters
Understanding and Improving Layer Normalization
Jingjing Xu, Xu Sun, Zhiyuan Zhang +2
Layer normalization (LayerNorm) is a technique to normalize the distributions of intermediate layers. It enables smoother gradients, faster training, and better generalization accu…
HighwayGraph: Modelling Long-distance Node Relations for Improving General Graph Neural Network
Deli Chen, Xiaoqian Liu, Yankai Lin +4
Graph Neural Networks (GNNs) are efficient approaches to process graph-structured data. Modelling long-distance node relations is essential for GNN training and applications. Howev…
An Adaptive and Momental Bound Method for Stochastic Learning
Jianbang Ding, Xuancheng Ren, Ruixuan Luo +1
Training deep neural networks requires intricate initialization and careful selection of learning rates. The emergence of stochastic gradient optimization methods that use adaptive…
Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View
Deli Chen, Yankai Lin, Wei Li +3
Graph Neural Networks (GNNs) have achieved promising performance on a wide range of graph-based tasks. Despite their success, one severe limitation of GNNs is the over-smoothing is…
Memorized Sparse Backpropagation
Zhiyuan Zhang, Pengcheng Yang, Xuancheng Ren +2
Neural network learning is usually time-consuming since backpropagation needs to compute full gradients and backpropagate them across multiple layers. Despite its success of existi…
Adaptive Gradient Methods with Dynamic Bound of Learning Rate
Liangchen Luo, Yuanhao Xiong, Yan Liu +1
Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been proposed to achieve a rapid training process with an element-wise scaling term on learning rates. Though p…