10 papers
Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
Jacqueline He, Howard Yen, Shuyue Stella Li +9
Logit-based knowledge distillation (KD) is used to train smaller language models (LMs) via supervision from stronger teachers, but whether its benefits are consistent across traini…
DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
Jiarui Feng, Hanqing Zeng, Karish Grover +11
Mixture-of-Experts (MoE) models have become a leading approach for decoupling parameter count from computational cost in large language models, yet effectively scaling MoE performa…
Agentic Recommender System with Hierarchical Belief-State Memory
Xiang Shen, Yuhang Zhou, Yifan Wu +8
Memory-augmented LLM agents have advanced personalized recommendation, yet existing approaches universally adopt flat memory representations that conflate ephemeral signals with st…
ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuning
Ruizhong Qiu, Hanqing Zeng, Yinglong Xia +15
Low-rank adapters (LoRAs) are a parameter-efficient finetuning technique that injects trainable low-rank matrices into pretrained models to adapt them to new tasks. Mixture-of-LoRA…
Rethinking ANN-based Retrieval: Multifaceted Learnable Index for Large-scale Recommendation System
Jiang Zhang, Yubo Wang, Wei Chang +14
Approximate nearest neighbor (ANN) search is widely used in the retrieval stage of large-scale recommendation systems. In this stage, candidate items are indexed using their learne…
Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation
Benyu Zhang, Qiang Zhang, Jianpeng Cheng +10
Large Language Models (LLMs) represent a promising frontier for recommender systems, yet their development has been impeded by the absence of predictable scaling laws, which are cr…
Efficient Sequential Recommendation for Long Term User Interest Via Personalization
Qiang Zhang, Hanchao Yu, Ivan Ji +14
Recent years have witnessed success of sequential modeling, generative recommender, and large language model for recommendation. Though the scaling law has been validated for seque…
RecoWorld: Building Simulated Environments for Agentic Recommender Systems
Fei Liu, Xinyu Lin, Hanchao Yu +12
We present RecoWorld, a blueprint for building simulated environments tailored to agentic recommender systems. Such environments give agents a proper training space where they can…
GEM: Empowering LLM for both Embedding Generation and Language Understanding
Caojin Zhang, Qiang Zhang, Ke Li +6
Large decoder-only language models (LLMs) have achieved remarkable success in generation and reasoning tasks, where they generate text responses given instructions. However, many a…
S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning
Hanqing Zeng, Yinglong Xia, Zhuokai Zhao +7
Fine-tuning pre-trained large language models (LLMs) presents a dual challenge of balancing parameter efficiency and model capacity. Existing methods like low-rank adaptations (LoR…