1 citations · 1 across the 6 of their papers we have counts for
6 papers
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
Xiaolong Li, Zhangchen Zhou, Zhi-Qin John Xu
Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable. W…
Towards Reliable Pediatric Brain Tumor Segmentation: Task-Specific nnU-Net Enhancements
Xiaolong Li, Zhi-Qin John Xu, Yan Ren +2
Accurate segmentation of pediatric brain tumors in multi-parametric magnetic resonance imaging (mpMRI) is critical for diagnosis, treatment planning, and monitoring, yet faces uniq…
Neural Network Based Framework for Passive Intermodulation Cancellation in MIMO Systems
Xiaolong Li, Zhi-Qin John Xu, Peiting You +1
Passive intermodulation (PIM) has emerged as a critical source of self-interference in modern MIMO-OFDM systems, especially under the stringent requirements of 5G and beyond. Conve…
Adaptive Preconditioners Trigger Loss Spikes in Adam
Zhiwei Bai, Zhangchen Zhou, Jiajie Zhao +6
Loss spikes commonly emerge during neural network training with the Adam optimizer across diverse architectures and scales, yet their underlying mechanism remains elusive. While pr…
Anchor function: a type of benchmark functions for studying language models
Zhongwang Zhang, Zhiwei Wang, Junjie Yao +4
Understanding transformer-based language models is becoming increasingly crucial, particularly as they play pivotal roles in advancing towards artificial general intelligence. Howe…
Loss Spike in Training Neural Networks
Xiaolong Li, Zhi-Qin John Xu, Zhongwang Zhang
In this work, we investigate the mechanism underlying loss spikes observed during neural network training. When the training enters a region with a lower-loss-as-sharper (LLAS) str…