2 papers
cs.LG2026
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Xingyue Huang, Xueying Ding, Mingxuan Ju +3
Softmax attention struggles with long contexts due to structural limitations: the strict sum-to-one constraint forces attention sinks on irrelevant tokens, and probability mass dis…
cs.CL2025
Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings
Xueying Ding, Xingyue Huang, Mingxuan Ju +5
Large language models produce powerful text embeddings, but their causal attention mechanism restricts the flow of information from later to earlier tokens, degrading representatio…