7 papers
Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration
Ruochen Jin, Zhanliang Wang, Zongyu Dai +2
Preference alignment often makes large language models (LLMs) overconfident and poorly calibrated. Traditional post-hoc temperature scaling is inherently domain-dependent: a temper…
RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization
Shenyang Deng, Zhuoli Ouyang, Tianyu Pang +4
Preconditioned adaptive methods have gained significant attention for training deep neural networks, as they capture rich curvature information of the loss landscape. The central c…
A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering
Zhanliang Wang, Jiancong Xiao, Ruochen Jin +3
Calibration measures whether a model's predicted confidence aligns with its empirical accuracy, and is central to the reliable deployment of large language models (LLMs) in high-st…
Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
Jiancong Xiao, Bojian Hou, Zhanliang Wang +4
One of the key technologies for the success of Large Language Models (LLMs) is preference alignment. However, a notable side effect of preference alignment is poor calibration: whi…
MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance
Jia Xu, Tianyi Wei, Bojian Hou +7
We introduce MentalChat16K, an English benchmark dataset combining a synthetic mental health counseling dataset and a dataset of anonymized transcripts from interventions between B…
ICAFS: Inter-Client-Aware Feature Selection for Vertical Federated Learning
Ruochen Jin, Boning Tong, Shu Yang +2
Vertical federated learning (VFL) enables a paradigm for vertically partitioned data across clients to collaboratively train machine learning models. Feature selection (FS) plays a…