9 papers
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Xingyue Huang, Xueying Ding, Mingxuan Ju +3
Softmax attention struggles with long contexts due to structural limitations: the strict sum-to-one constraint forces attention sinks on irrelevant tokens, and probability mass dis…
Heuristic Methods are Good Teachers to Distill MLPs for Graph Link Prediction
Zongyue Qin, Shichang Zhang, Mingxuan Ju +3
Link prediction is a crucial graph-learning task with applications including citation prediction and product recommendation. Distilling Graph Neural Networks (GNNs) teachers into M…
Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings
Xueying Ding, Xingyue Huang, Mingxuan Ju +5
Large language models produce powerful text embeddings, but their causal attention mechanism restricts the flow of information from later to earlier tokens, degrading representatio…
On the Role of Weight Decay in Collaborative Filtering: A Popularity Perspective
Donald Loveland, Mingxuan Ju, Tong Zhao +2
Collaborative filtering (CF) enables large-scale recommendation systems by encoding information from historical user-item interactions into dense ID-embedding tables. However, as e…
Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning
Ngoc Bui, Menglin Yang, Runjin Chen +5
Backward compatible representation learning enables updated models to integrate seamlessly with existing ones, avoiding to reprocess stored data. Despite recent advances, existing…
Beyond Unimodal Boundaries: Generative Recommendation with Multimodal Semantics
Jing Zhu, Mingxuan Ju, Yozen Liu +3
Generative recommendation (GR) has become a powerful paradigm in recommendation systems that implicitly links modality and semantics to item representation, in contrast to previous…