4 papers
Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel
Chuanyang Zheng, Jiankai Sun, Yihang Gao +13
Mixture-of-Experts (MoE) has become a cornerstone in recent state-of-the-art large language models (LLMs). Traditionally, MoE relies on as the router score funct…
Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning
Peihao Wang, Zhangyang Wang
We develop a theoretical framework that explains how discrete symbolic structures can emerge naturally from continuous neural network training dynamics. By lifting neural parameter…
Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models
Haoyu Wang, Peihao Wang, Mufei Li +4
Modern large language models (LLMs) are inherently auto-regressive, requiring input to be serialized into flat sequences regardless of their structural dependencies. This serializa…
Oscillation Inversion: Understand the structure of Large Flow Model through the Lens of Inversion Method
Yan Zheng, Zhenxiao Liang, Xiaoyan Cong +4
We explore the oscillatory behavior observed in inversion methods applied to large-scale text-to-image diffusion models, with a focus on the "Flux" model. By employing a fixed-poin…