7 citations · 11 across the 11 of their papers we have counts for
9 papers · 1 filter
The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics
Yihe Zhou, Tongtian Zhu, Yingxiao Huo +4
Adam is the default optimizer for training modern deep neural networks, yet its adaptive behavior remains poorly understood due to the complex interaction between its first- and se…
Mobius Learning: Cyclic Depth Folding in Transformers
Tongtian Zhu
Transformer-based language models organize computation along an ordered depth axis, where shallow and deep blocks often develop distinct representational roles. We challenge the co…
DICE: Data Influence Cascade in Decentralized Learning
Tongtian Zhu, Wenhao Li, Can Wang +1
Decentralized learning offers a promising approach to crowdsource data consumptions and computational workloads across geographically distributed compute interconnected through pee…
On the Surprising Effectiveness of a Single Global Merging in Decentralized Learning
Tongtian Zhu, Tianyu Zhang, Mingze Wang +2
Decentralized learning provides a scalable alternative to parameter-server-based training, yet its performance is often hindered by limited peer-to-peer communication. In this pape…
Adversarial Erasing with Pruned Elements: Towards Better Graph Lottery Ticket
Yuwen Wang, Shunyu Liu, Kaixuan Chen +5
Graph Lottery Ticket (GLT), a combination of core subgraph and sparse subnetwork, has been proposed to mitigate the computational cost of deep Graph Neural Networks (GNNs) on large…
Improving Expressivity of GNNs with Subgraph-specific Factor Embedded Normalization
Kaixuan Chen, Shunyu Liu, Tongtian Zhu +5
Graph Neural Networks (GNNs) have emerged as a powerful category of learning architecture for handling graph-structured data. However, existing GNNs typically ignore crucial struct…
Decentralized SGD and Average-direction SAM are Asymptotically Equivalent
Tongtian Zhu, Fengxiang He, Kaixuan Chen +2
Decentralized stochastic gradient descent (D-SGD) allows collaborative learning on massive devices simultaneously without the control of a central server. However, existing theorie…
Contrastive Identity-Aware Learning for Multi-Agent Value Decomposition
Shunyu Liu, Yihe Zhou, Jie Song +5
Value Decomposition (VD) aims to deduce the contributions of agents for decentralized policies in the presence of only global rewards, and has recently emerged as a powerful credit…
Topology-aware Generalization of Decentralized SGD
Tongtian Zhu, Fengxiang He, Lan Zhang +3
This paper studies the algorithmic stability and generalizability of decentralized stochastic gradient descent (D-SGD). We prove that the consensus model learned by D-SGD is $\math…