Publications (27)
Finding Skill Neurons in Pre-trained Transformer-based Language Models
Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang +3
Configuration-to-Performance Scaling Law with Neural Ansatz
Huaqing Zhang, Kaiyue Wen, Tengyu Ma
Residual permutation test for regression coefficient testing
Kaiyue Wen, Tengyao Wang, Yuhao Wang
Weight Ensembling Improves Reasoning in Language Models
Xingyu Dang, Christina Baek, Kaiyue Wen +2
Training Hybrid Block Diffusion Language Models with Partial Bidirectionality
Pranshu Chaturvedi, Parth Shroff, Tarun Suresh +2
Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning
Arvind Mahankali, Kaiyue Wen, Tengyu Ma
Task Generalization With AutoRegressive Compositional Structure: Can Learning From $D$ Tasks Generalize to $D^{T}$ Tasks?
Amirhesam Abedsoltan, Huaqing Zhang, Kaiyue Wen +3
Overtrained Language Models Are Harder to Fine-Tune
Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen +5
On Transferability of Prompt Tuning for Natural Language Processing
Yusheng Su, Xiaozhi Wang, Yujia Qin +10
QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation
Jiazheng Li, Hongzhou Lin, Hong Lu +5
Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
Zihan Qiu, Zeyu Huang, Bo Zheng +7
Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars
Kaiyue Wen, Yuchen Li, Bingbin Liu +1
Benign Overfitting in Classification: Provably Counter Label Noise with Larger Models
Kaiyue Wen, Jiaye Teng, Jingzhao Zhang
How Does Sharpness-Aware Minimization Minimize Sharpness?
Kaiyue Wen, Tengyu Ma, Zhiyuan Li
Practically Solving LPN in High Noise Regimes Faster Using Neural Networks
Haozhe Jiang, Kaiyue Wen, Yilei Chen
A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
Zihan Qiu, Zeyu Huang, Kaiyue Wen +16
PaTH Attention: Position Encoding via Accumulating Householder Transformations
Songlin Yang, Yikang Shen, Kaiyue Wen +5
From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency
Kaiyue Wen, Huaqing Zhang, Hongzhou Lin +1
RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval
Kaiyue Wen, Xingyu Dang, Kaifeng Lyu
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Zihan Qiu, Zekun Wang, Bo Zheng +10
Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
Kaiyue Wen, Zhiyuan Li, Jason Wang +3
Scaling Self-Play with Self-Guidance
Luke Bailey, Kaiyue Wen, Kefan Dong +2
Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization
Kaiyue Wen, Zhiyuan Li, Tengyu Ma
Pseudo-Formalization for Automatic Proof Verification
Slim Barkallah, Luke Bailey, Kaiyue Wen +2
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2
Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images
Shengguang Wu, Fan-Yun Sun, Kaiyue Wen +1
Fantastic Pretraining Optimizers and Where to Find Them
Kaiyue Wen, David Hall, Tengyu Ma +1