Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization

Zhehang Du, Hangfeng He, Weijie Su

Large language models (LLMs) are pretrained by minimizing the cross-entropy loss for next-token prediction. In this paper, we study whether this optimization strategy can induce ge…