8 papers
Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specialization
Rizhen Hu, Yuan Cao, Boao Kong +2
Sparse Mixture-of-Experts (MoE) models scale Transformers efficiently but suffer from expert overlap -- redundant representations across experts and routing ambiguity, resulting in…
GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training
Yuan Cao, Dezhi Ran, Mengzhou Wu +9
Post-training GUI agents in interactive environments is critical for developing generalization and long-horizon planning capabilities. However, training on real-world applications…
From Classification to Ranking: Enhancing LLM Reasoning Capabilities for MBTI Personality Detection
Yuan Cao, Feixiang Liu, Xinyue Wang +4
Personality detection aims to measure an individual's corresponding personality traits through their social media posts. The advancements in Large Language Models (LLMs) offer nove…
Reasoning over Precedents Alongside Statutes: Case-Augmented Deliberative Alignment for LLM Safety
Can Jin, Rui Wu, Tong Che +10
Ensuring that Large Language Models (LLMs) adhere to safety principles without refusing benign requests remains a significant challenge. While OpenAI introduces deliberative alignm…
Towards Understanding Transformers in Learning Random Walks
Wei Shi, Yuan Cao
Transformers have proven highly effective across various applications, especially in handling sequential data such as natural languages and time series. However, transformer models…
KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Bandits
Dezhi Ran, Shuxiao Xie, Mingfang Ji +9
High-performance GPU kernels are critical for efficient LLM serving, yet their optimization remains a bottleneck requiring deep system expertise. While code LLMs show promise in ge…
TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Network
Guangxin He, Yuan Cao, Yutong He +4
Decentralized training of large language models offers the opportunity to pool computational resources across geographically distributed participants, but is often bottlenecked by…
Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall
Qianli Wang, Mingyang Wang, Nils Feldhus +5
Quantization methods are widely used to accelerate inference and streamline the deployment of large language models (LLMs). Although quantization's effects on various LLM capabilit…