7 papers
MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
Bohan Lyu, Yucheng Yang, Siqiao Huang +25
Modern AI progress has been driven by ML methods that are generalizable across settings and scalable to larger regimes. As large language models demonstrate advanced capabilities i…
Zenith: Scaling up Ranking Models for Billion-scale Livestreaming Recommendation
Ruifeng Zhang, Zexi Huang, Zikai Wang +11
Accurately capturing feature interactions is essential in recommender systems, and recent trends show that scaling up model capacity could be a key driver for next-level predictive…
SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
Wentao Guo, Mayank Mishra, Xinle Cheng +2
Mixture of Experts (MoE) models have emerged as the de facto architecture for scaling up language models without significantly increasing the computational cost. Recent MoE models…
Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity
Yide Ran, Wentao Guo, Jingwei Sun +7
Federated Learning enables collaborative fine-tuning of Large Language Models (LLMs) across decentralized Non-Independent and Identically Distributed (Non-IID) clients, but such mo…
Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models
Yuheng Wu, Wentao Guo, Zirui Liu +3
This paper investigates the emergence of Theory-of-Mind (ToM) capabilities in large language models (LLMs) from a mechanistic perspective, focusing on the role of extremely sparse…
Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity
Wentao Guo, Jikai Long, Yimeng Zeng +9
Zeroth-order optimization (ZO) is a memory-efficient strategy for fine-tuning Large Language Models using only forward passes. However, the application of ZO fine-tuning in memory-…
Ranking with Slot Constraints
Wentao Guo, Andrew Wang, Bradon Thymes +1
We introduce the problem of ranking with slot constraints, which can be used to model a wide range of application problems -- from college admission with limited slots for differen…