2 citations · 2 across the 8 of their papers we have counts for
7 papers · 1 filter
Skip a Layer or Loop It? Learning Program-of-Layers in LLMs
Ziyue Li, Yang Li, Tianyi Zhou
Large language models (LLMs) perform inference by following a fixed depth and order, non-recurrent execution of all layers. We reveal the wide existence of training-free, flexible,…
Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs
Zhongyang Li, Ziyue Li, Tianyi Zhou
Sparse Mixture-of-Experts (MoE) have been widely adopted in recent large language models since it can efficiently scale up the model capability without increasing the inference cos…
Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs
Ziyue Li, Yang Li, Tianyi Zhou
Can a pretrained neural network adapt its architecture to different inputs without any finetuning? Do we need all layers for simple tasks, and are they adequate for challenging tas…
Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Test
Ziyue Li, Chenrui Fan, Tianyi Zhou
This paper presents the first study of grokking in practical LLM pretraining. Specifically, we investigate when an LLM memorizes the training data, when its generalization on downs…
C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization for Test-Time Expert Re-Mixing
Zhongyang Li, Ziyue Li, Tianyi Zhou
Mixture-of-Experts (MoE) Large Language Models (LLMs) suffer from severely sub-optimal expert pathways-our study reveals that naive expert selection learned from pretraining leaves…
How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients
Ming Li, Yanhong Li, Ziyue Li +1
As the post-training of large language models (LLMs) advances from instruction-following to complex reasoning tasks, understanding how different data affect finetuning dynamics rem…