6 citations · 6 across the 7 of their papers we have counts for
8 papers
Redesign Mixture-of-Experts Routers with Manifold Power Iteration
Songhao Wu, Ang Lv, Ruobing Xie +1
Router is the cornerstone component to the Mixture-of-Experts models. Serving as expert proxies, the rows of the router matrix compute their similarity to the MoE inputs to determi…
Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings
Songhao Wu, Zhongxin Chen, Yuxuan Liu +3
Large language models exhibit impressive zero-shot capabilities across a wide range of downstream tasks. However, they struggle to function as off-the-shelf embedding models, leadi…
Bridge the Gap between Past and Future: Siamese Model Optimization for Context-Aware Document Ranking
Songhao Wu, Quan Tu, Mingjie Zhong +4
In the realm of information retrieval, users often engage in multi-turn interactions with search engines to acquire information, leading to the formation of sequences of user feedb…
Unify Graph Learning with Text: Unleashing LLM Potentials for Session Search
Songhao Wu, Quan Tu, Hong Liu +6
Session search involves a series of interactive queries and actions to fulfill user's complex information need. Current strategies typically prioritize sequential modeling for deep…
From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
Jia-Nan Li, Jian Guan, Songhao Wu +2
Large language models (LLMs) have traditionally been aligned through one-size-fits-all approaches that assume uniform human preferences, fundamentally overlooking the diversity in…
PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration
Songhao Wu, Ang Lv, Xiao Feng +5
The KV cache in large language models is a dominant factor in memory usage, limiting their broader applicability. Quantizing the cache to lower bit widths is an effective way to re…
Autonomy-of-Experts Models
Ang Lv, Ruobing Xie, Yining Qian +5
Mixture-of-Experts (MoE) models mostly use a router to assign tokens to specific expert modules, activating only partial parameters and often outperforming dense models. We argue t…
PEAR: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference Overhead
Tao Tan, Yining Qian, Ang Lv +7
Large language models (LLMs) enhanced with retrieval-augmented generation (RAG) have introduced a new paradigm for web search. However, the limited context awareness of LLMs degrad…