12 papers
BEAR: Towards Beam-Search-Aware Optimization for Recommendation with Large Language Models
Weiqin Yang, Bohao Wang, Zhenxiang Xu +5
Recent years have seen a rapid surge in research leveraging Large Language Models (LLMs) for recommendation. These methods typically employ supervised fine-tuning (SFT) to adapt LL…
When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework
Zhen Xu, Shang Zhu, Jue Wang +5
We investigate the challenge of applying Large Language Models (LLMs) to long texts. We propose a theoretical framework that distinguishes the failure modes of long context tasks i…
From Data to Behavior: Predicting Unintended Model Behaviors Before Training
Mengru Wang, Zhenqian Xu, Junfeng Fang +4
Large Language Models (LLMs) can acquire unintended biases from seemingly benign training data even without explicit cues or malicious content. Existing methods struggle to detect…
Memorization Dynamics in Knowledge Distillation for Language Models
Jaydeep Borkar, Karan Chadha, Niloofar Mireshghallah +6
Knowledge Distillation (KD) is increasingly adopted to transfer capabilities from large language models to smaller ones, offering significant improvements in efficiency and utility…
Enhancing LLM-Based Data Annotation with Error Decomposition
Zhen Xu, Vedant Khatri, Yijun Dai +4
Large language models offer a scalable alternative to human coding for data annotation tasks, enabling the scale-up of research across data-intensive domains. While LLMs are alread…
Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder
Zhen Xu, Zhen Tan, Song Wang +2
Sparse autoencoders (SAEs) have emerged as a powerful tool for interpreting large language models (LLMs) by decomposing token activations into combinations of human-understandable…