1 citations · 1 across the 8 of their papers we have counts for
9 papers
X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths
Bowen Dong, Yilong Fan, Tengyu Pan +6
Mixture-of-Depths (MoD) enables conditional computation across Transformer depth by routing only a subset of tokens through selected layers, but its original one-sparse--one-dense…
CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution
Teng Pan, Yuchen Yan, Zixuan Wang +6
Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as…
Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings
Tengyu Pan, Zhichao Duan, Zhenyu Li +4
Text embedding models are essential for various natural language processing tasks, enabling the effective encoding of semantic information into dense vector representations. These…
Maximum Score Routing For Mixture-of-Experts
Bowen Dong, Yilong Fan, Yutao Sun +4
Routing networks in sparsely activated mixture-of-experts (MoE) dynamically allocate input tokens to top-k experts through differentiable sparse transformations, enabling scalable…
Efficient Attention Mechanisms for Large Language Models: A Survey
Yutao Sun, Zhenyu Li, Yike Zhang +4
Transformer-based architectures have become the prevailing backbone of large language models. However, the quadratic time and memory complexity of self-attention remains a fundamen…
COMM:Concentrated Margin Maximization for Robust Document-Level Relation Extraction
Zhichao Duan, Tengyu Pan, Zhenyu Li +2
Document-level relation extraction (DocRE) is the process of identifying and extracting relations between entities that span multiple sentences within a document. Due to its realis…
FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling
Weilin Zhao, Tengyu Pan, Xu Han +9
Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models (LLMs) by utilizing a draft-then-verify…
TROI: Cross-Subject Pretraining with Sparse Voxel Selection for Enhanced fMRI Visual Decoding
Ziyu Wang, Tengyu Pan, Zhenyu Li +3
fMRI (functional Magnetic Resonance Imaging) visual decoding involves decoding the original image from brain signals elicited by visual stimuli. This often relies on manually label…
FocusLLM: Precise Understanding of Long Context by Dynamic Condensing
Zhenyu Li, Yike Zhang, Tengyu Pan +6
Empowering LLMs with the ability to precisely understand long contexts is crucial for many downstream applications. However, handling long contexts with conventional transformer ar…