1 citations · 1 across the 4 of their papers we have counts for
5 papers
Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Ermo Hua, Xiangyu Hong, Baiting Wu +11
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchica…
Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning
Kai Chen, Jifeng Ding, Ning Ding +44
We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve comp…
DePass: Unified Feature Attributing by Simple Decomposed Forward Pass
Xiangyu Hong, Che Jiang, Kai Tian +4
Attributing the behavior of Transformer models to internal computations is a central challenge in mechanistic interpretability. We introduce DePass, a unified framework for feature…
On the token distance modeling ability of higher RoPE attention dimension
Xiangyu Hong, Che Jiang, Biqing Qi +4
Length extrapolation algorithms based on Rotary position embedding (RoPE) have shown promising results in extending the context length of language models. However, understanding ho…
On Large Language Models' Hallucination with Regard to Known Facts
Che Jiang, Biqing Qi, Xiangyu Hong +6
Large language models are successful in answering factoid questions but are also prone to hallucination. We investigate the phenomenon of LLMs possessing correct answer knowledge y…