5 citations · 14 across the 12 of their papers we have counts for
6 papers · 1 filter
OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling
Yuxuan Lou, Yang You
Muon fixes the \emph{direction} of every matrix-valued update at the polar factor of its momentum, while each layer's step \emph{magnitude} is addressed only by a static shape corr…
Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning
Peihao Wang, Shan Yang, Xijun Wang +8
Associative memory has long underpinned the design of sequential models. Beyond recall, humans reason by projecting future states and selecting goal-directed actions, a capability…
MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
Geng Zhang, Yuxuan Han, Yuxuan Lou +3
Mixture-of-Experts (MoE) enables efficient scaling of large language models by activating only a subset of experts per input token. However, deploying MoE-based models incurs signi…
RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents
Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou +6
Owing to recent advancements, Large Language Models (LLMs) can now be deployed as agents for increasingly complex decision-making applications in areas including robotics, gaming,…
One Student Knows All Experts Know: From Sparse to Dense
Fuzhao Xue, Xiaoxin He, Xiaozhe Ren +2
Human education system trains one student by multiple experts. Mixture-of-experts (MoE) is a powerful sparse architecture including multiple experts. However, sparse MoE model is e…
Cross-token Modeling with Conditional Computation
Yuxuan Lou, Fuzhao Xue, Zangwei Zheng +1
Mixture-of-Experts (MoE), a conditional computation architecture, achieved promising performance by scaling local module (i.e. feed-forward network) of transformer. However, scalin…