4 papers · 1 filter
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Xingyue Huang, Xueying Ding, Mingxuan Ju +3
Softmax attention struggles with long contexts due to structural limitations: the strict sum-to-one constraint forces attention sinks on irrelevant tokens, and probability mass dis…
Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning
Ngoc Bui, Menglin Yang, Runjin Chen +5
Backward compatible representation learning enables updated models to integrate seamlessly with existing ones, avoiding to reprocess stored data. Despite recent advances, existing…
Heuristic Methods are Good Teachers to Distill MLPs for Graph Link Prediction
Zongyue Qin, Shichang Zhang, Mingxuan Ju +3
Link prediction is a crucial graph-learning task with applications including citation prediction and product recommendation. Distilling Graph Neural Networks (GNNs) teachers into M…
Enhancing Item Tokenization for Generative Recommendation through Self-Improvement
Runjin Chen, Mingxuan Ju, Ngoc Bui +7
Generative recommendation systems, driven by large language models (LLMs), present an innovative approach to predicting user preferences by modeling items as token sequences and ge…