9 citations · 9 across the 4 of their papers we have counts for
5 papers
X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths
Bowen Dong, Yilong Fan, Tengyu Pan +6
Mixture-of-Depths (MoD) enables conditional computation across Transformer depth by routing only a subset of tokens through selected layers, but its original one-sparse--one-dense…
Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings
Tengyu Pan, Zhichao Duan, Zhenyu Li +4
Text embedding models are essential for various natural language processing tasks, enabling the effective encoding of semantic information into dense vector representations. These…
Maximum Score Routing For Mixture-of-Experts
Bowen Dong, Yilong Fan, Yutao Sun +4
Routing networks in sparsely activated mixture-of-experts (MoE) dynamically allocate input tokens to top-k experts through differentiable sparse transformations, enabling scalable…
Efficient Attention Mechanisms for Large Language Models: A Survey
Yutao Sun, Zhenyu Li, Yike Zhang +4
Transformer-based architectures have become the prevailing backbone of large language models. However, the quadratic time and memory complexity of self-attention remains a fundamen…
FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering
Zhenyu Li, Sunqi Fan, Yu Gu +5
Knowledge base question answering (KBQA) is a critical yet challenging task due to the vast number of entities within knowledge bases and the diversity of natural language question…