7 papers
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Xingyue Huang, Xueying Ding, Mingxuan Ju +3
Softmax attention struggles with long contexts due to structural limitations: the strict sum-to-one constraint forces attention sinks on irrelevant tokens, and probability mass dis…
Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings
Xueying Ding, Xingyue Huang, Mingxuan Ju +5
Large language models produce powerful text embeddings, but their causal attention mechanism restricts the flow of information from later to earlier tokens, degrading representatio…
Node Duplication Improves Cold-start Link Prediction
Zhichun Guo, Tong Zhao, Yozen Liu +5
Graph Neural Networks (GNNs) are prominent in graph machine learning and have shown state-of-the-art performance in Link Prediction (LP) tasks. Nonetheless, recent studies show tha…
GiGL: Large-Scale Graph Neural Networks at Snapchat
Tong Zhao, Yozen Liu, Matthew Kolodner +12
Recent advances in graph machine learning (ML) with the introduction of Graph Neural Networks (GNNs) have led to a widespread interest in applying these approaches to business appl…
Beyond Unimodal Boundaries: Generative Recommendation with Multimodal Semantics
Jing Zhu, Mingxuan Ju, Yozen Liu +3
Generative recommendation (GR) has become a powerful paradigm in recommendation systems that implicitly links modality and semantics to item representation, in contrast to previous…
GraphHash: Graph Clustering Enables Parameter Efficiency in Recommender Systems
Xinyi Wu, Donald Loveland, Runjin Chen +7
Deep recommender systems rely heavily on large embedding tables to handle high-cardinality categorical features such as user/item identifiers, and face significant memory constrain…