8 citations · 8 across the 8 of their papers we have counts for
8 papers
SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging
Zijun Chen, Zhanpeng Zhou, Bo Zhang +3
Model merging has gained increasing attention due to its intriguing property: interpolating the parameters of different task-specific fine-tuned models leads to multi-task abilitie…
New Evidence of the Two-Phase Learning Dynamics of Neural Networks
Zhanpeng Zhou, Yongyi Yang, Mahito Sugiyama +1
Understanding how deep neural networks learn remains a fundamental challenge in modern machine learning. A growing body of evidence suggests that training dynamics undergo a distin…
On the Cone Effect in the Learning Dynamics
Zhanpeng Zhou, Yongyi Yang, Jie Ren +2
Understanding the learning dynamics of neural networks is a central topic in the deep learning community. In this paper, we take an empirical perspective to study the learning dyna…
On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm
Zhanpeng Zhou, Zijun Chen, Yilan Chen +2
The pretraining-finetuning paradigm has become the prevailing trend in modern deep learning. In this work, we discover an intriguing linear phenomenon in models that are initialize…
Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory
Yiting Chen, Zhanpeng Zhou, Junchi Yan
The behavior of neural networks still remains opaque, and a recently widely noted phenomenon is that networks often achieve similar performance when initialized with different rand…
Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity
Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang +2
Recent work has revealed many intriguing empirical phenomena in neural network training, despite the poorly understood and highly complex loss landscapes and training dynamics. One…
Defects of Convolutional Decoder Networks in Frequency Representation
Ling Tang, Wen Shen, Zhanpeng Zhou +2
In this paper, we prove the representation defects of a cascaded convolutional decoder network, considering the capacity of representing different frequency components of an input…
Batch Normalization Is Blind to the First and Second Derivatives of the Loss
Zhanpeng Zhou, Wen Shen, Huixin Chen +2
In this paper, we prove the effects of the BN operation on the back-propagation of the first and second derivatives of the loss. When we do the Taylor series expansion of the loss…