3 papers
cs.LG2025
Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch
Wen-Shu Fan, Xin-Chun Li, De-Chuan Zhan
Knowledge Distillation (KD) could transfer the ``dark knowledge" of a well-performed yet large neural network to a weaker but lightweight one. From the view of output logits and so…
cs.LG2025
Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks
Yichu Xu, Xin-Chun Li, Lan Li +1
The loss landscape of deep neural networks (DNNs) is commonly considered complex and wildly fluctuated. However, an interesting observation is that the loss surfaces plotted along…
cs.LG2024
Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks
Xin-Chun Li, Jin-Lin Tang, Bo Zhang +2
Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat…