8 papers
CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution
Teng Pan, Yuchen Yan, Zixuan Wang +6
Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as…
Efficient Attention Mechanisms for Large Language Models: A Survey
Yutao Sun, Zhenyu Li, Yike Zhang +4
Transformer-based architectures have become the prevailing backbone of large language models. However, the quadratic time and memory complexity of self-attention remains a fundamen…
Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings
Tengyu Pan, Zhichao Duan, Zhenyu Li +4
Text embedding models are essential for various natural language processing tasks, enabling the effective encoding of semantic information into dense vector representations. These…
Maximum Score Routing For Mixture-of-Experts
Bowen Dong, Yilong Fan, Yutao Sun +4
Routing networks in sparsely activated mixture-of-experts (MoE) dynamically allocate input tokens to top-k experts through differentiable sparse transformations, enabling scalable…
COMM:Concentrated Margin Maximization for Robust Document-Level Relation Extraction
Zhichao Duan, Tengyu Pan, Zhenyu Li +2
Document-level relation extraction (DocRE) is the process of identifying and extracting relations between entities that span multiple sentences within a document. Due to its realis…
FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling
Weilin Zhao, Tengyu Pan, Xu Han +9
Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models (LLMs) by utilizing a draft-then-verify…