6 papers · 1 filter
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
Tian Ding, Dawei Li, Ruoyu Sun
We investigate the geometric structure of stationary plateaus that arise in the loss landscape of two-layer neural networks with smooth activation functions. We focus on the phenom…
Learning to Gridize: Segment Physical World by Wireless Communication Channel
Juntao Wang, Feng Yin, Tian Ding +3
Gridization, the process of partitioning space into grids where users share similar channel characteristics, serves as a fundamental prerequisite for efficient large-scale network…
Exploring and Improving Initialization for Deep Graph Neural Networks: A Signal Propagation Perspective
Senmiao Wang, Yupeng Chen, Yushun Zhang +2
Graph Neural Networks (GNNs) often suffer from performance degradation as the network depth increases. This paper addresses this issue by introducing initialization methods that en…
MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
Yupeng Chen, Senmiao Wang, Yushun Zhang +5
Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. Typically, LLMs are first pre-trained on large corpora and subsequently fine-tu…
PDHG-Unrolled Learning-to-Optimize Method for Large-Scale Linear Programming
Bingheng Li, Linxin Yang, Yupeng Chen +8
Solving large-scale linear programming (LP) problems is an important task in various areas such as communication networks, power systems, finance and logistics. Recently, two disti…
Adam-mini: Use Fewer Learning Rates To Gain More
Yushun Zhang, Congliang Chen, Ziniu Li +6
We propose Adam-mini, an optimizer that achieves on par or better performance than AdamW with 50% less memory footprint. Adam-mini reduces memory by cutting down the learning rate…