3 citations · 12 across the 22 of their papers we have counts for
9 papers · 1 filter
More Expressive Attention with Negative Weights
Ang Lv, Ruobing Xie, Shuaipeng Li +5
We propose a novel attention mechanism, named Cog Attention, that enables attention weights to be negative for enhanced expressiveness, which stems from two key factors: (1) Cog At…
PEAR: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference Overhead
Tao Tan, Yining Qian, Ang Lv +7
Large language models (LLMs) enhanced with retrieval-augmented generation (RAG) have introduced a new paradigm for web search. However, the limited context awareness of LLMs degrad…
HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation
Yuhan Chen, Ang Lv, Jian Luan +2
Many positional encodings (PEs) are designed to exhibit long-term decay, based on an entrenched and long-standing inductive opinion: tokens farther away from the current position c…
Language Models "Grok" to Copy
Ang Lv, Ruobing Xie, Xingwu Sun +2
We examine the pre-training dynamics of language models, focusing on their ability to copy text from preceding context--a fundamental skill for various LLM applications, including…
Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules
Zhuocheng Gong, Ang Lv, Jian Guan +6
Is it always necessary to compute tokens from shallow to deep layers in Transformers? The continued success of vanilla Transformers and their variants suggests an undoubted "yes".…
Mixture of In-Context Experts Enhance LLMs' Long Context Awareness
Hongzhan Lin, Ang Lv, Yuhan Chen +4
Many studies have revealed that large language models (LLMs) exhibit uneven awareness of different contextual positions. Their limited context awareness can lead to overlooking cri…
Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Models
Ang Lv, Yuhan Chen, Kaiyi Zhang +5
In this paper, we delve into several mechanisms employed by Transformer-based language models (LLMs) for factual recall tasks. We outline a pipeline consisting of three major steps…
Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models
Changyu Chen, Xiting Wang, Ting-En Lin +6
In reasoning tasks, even a minor error can cascade into inaccurate results, leading to suboptimal performance of large language models in such domains. Earlier fine-tuning approach…
Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning
Kaiyi Zhang, Ang Lv, Yuhan Chen +3
In this paper, by treating in-context learning (ICL) as a meta-optimization process, we explain why LLMs are sensitive to the order of ICL examples. This understanding leads us to…