4 papers
Zenith: Scaling up Ranking Models for Billion-scale Livestreaming Recommendation
Ruifeng Zhang, Zexi Huang, Zikai Wang +11
Accurately capturing feature interactions is essential in recommender systems, and recent trends show that scaling up model capacity could be a key driver for next-level predictive…
SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
Wentao Guo, Mayank Mishra, Xinle Cheng +2
Mixture of Experts (MoE) models have emerged as the de facto architecture for scaling up language models without significantly increasing the computational cost. Recent MoE models…
Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity
Yide Ran, Wentao Guo, Jingwei Sun +7
Federated Learning enables collaborative fine-tuning of Large Language Models (LLMs) across decentralized Non-Independent and Identically Distributed (Non-IID) clients, but such mo…
Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models
Yuheng Wu, Wentao Guo, Zirui Liu +3
This paper investigates the emergence of Theory-of-Mind (ToM) capabilities in large language models (LLMs) from a mechanistic perspective, focusing on the role of extremely sparse…