29 citations · 43 across the 19 of their papers we have counts for
21 papers
Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics
Mengyuan Sun, Yu Li, Zhuohao Yu +2
Rubric-based evaluation is a promising paradigm for judging large language model (LLM) outputs, yet self-generated rubrics lag human-annotated criteria on hard instances. We argue…
From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism
Zhuohao Yu, Zhiwei Steven Wu, Adam Block
Inference-time compute scaling has emerged as a powerful paradigm for improving language model performance on a wide range of tasks, but the question of how best to use the additio…
SteerRM: Debiasing Reward Models via Sparse Autoencoders
Mengyuan Sun, Zhuohao Yu, Weizheng Gu +2
Reward models (RMs) are critical components of alignment pipelines, yet they exhibit biases toward superficial stylistic cues, preferring better-presented responses over semantical…
What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?
Weizheng Gu, Chengze Li, Zhuohao Yu +6
Large language models are increasingly evaluated as interactive agents, yet standard agent benchmarks conflate two qualitatively distinct sources of success: semantic tool-use and…
Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning
Jiahao Zhang, Lujing Zhang, Keltin Grimes +3
A recurring challenge in preference fine-tuning (PFT) is handling (i.e., cyclic) preferences. Intransitive preferences often stem from either …
TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
Yidong Wang, Yunze Song, Tingyuan Zhu +11
The adoption of Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks. We identify two fundam…
SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation
Zhengran Zeng, Ruikai Shi, Keke Han +7
Automated Code Review (ACR) is crucial for software quality, yet existing benchmarks often fail to reflect real-world complexities, hindering the evaluation of modern Large Languag…
SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders
Zhuohao Yu, Xingru Jiang, Weizheng Gu +4
Watermarking LLM-generated text is critical for content attribution and misinformation prevention. However, existing methods compromise text quality, require white-box model access…
RewardAnything: Generalizable Principle-Following Reward Models
Zhuohao Yu, Jiali Zeng, Weizheng Gu +7
Reward Models, essential for guiding Large Language Model optimization, are typically trained on fixed preference datasets, resulting in rigid alignment to single, implicit prefere…
An Empirical Analysis of Uncertainty in Large Language Model Evaluations
Qiujie Xie, Qingqiu Li, Zhuohao Yu +3
As LLM-as-a-Judge emerges as a new paradigm for assessing large language models (LLMs), concerns have been raised regarding the alignment, bias, and stability of LLM evaluators. Wh…
Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Zhuohao Yu, Weizheng Gu, Yidong Wang +5
Large Language Models excel at code generation yet struggle with complex programming tasks that demand sophisticated reasoning. To bridge this gap, traditional process supervision…
FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Models
Zhuohao Yu, Chang Gao, Wenjin Yao +6
The rapid development of large language model (LLM) evaluation methodologies and datasets has led to a profound challenge: integrating state-of-the-art evaluation techniques cost-e…
E2ETune: End-to-End Knob Tuning via Fine-tuned Generative Language Model
Xinmei Huang, Haoyang Li, Jing Zhang +7
Database knob tuning is a significant challenge for database administrators, as it involves tuning a large number of configuration knobs with continuous or discrete values to achie…
CodeShell Technical Report
Rui Xie, Zhengran Zeng, Zhuohao Yu +3
Code large language models mark a pivotal breakthrough in artificial intelligence. They are specifically crafted to understand and generate programming languages, significantly boo…
KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models
Zhuohao Yu, Chang Gao, Wenjin Yao +6
Automatic evaluation methods for large language models (LLMs) are hindered by data contamination, leading to inflated assessments of their effectiveness. Existing strategies, which…
Supervised Knowledge Makes Large Language Models Better In-context Learners
Linyi Yang, Shuibai Zhang, Zhuohao Yu +8
Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The recent progress in large-scale generative models has further expanded th…
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
Yidong Wang, Zhuohao Yu, Zhengran Zeng +10
Instruction tuning large language models (LLMs) remains a challenging task, owing to the complexity of hyperparameter selection and the difficulty involved in evaluating the tuned…
Exploring Vision-Language Models for Imbalanced Learning
Yidong Wang, Zhuohao Yu, Jindong Wang +6
Vision-Language models (VLMs) that use contrastive language-image pre-training have shown promising zero-shot classification performance. However, their performance on imbalanced d…
TextBox 2.0: A Text Generation Library with Pre-trained Language Models
Tianyi Tang, Junyi Li, Zhipeng Chen +9
To facilitate research on text generation, this paper presents a comprehensive and unified library, TextBox 2.0, focusing on the use of pre-trained language models (PLMs). To be co…
ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models
Junyi Li, Tianyi Tang, Zheng Gong +6
Nowadays, pretrained language models (PLMs) have dominated the majority of NLP tasks. While, little research has been conducted on systematically evaluating the language abilities…
TextBox: A Unified, Modularized, and Extensible Framework for Text Generation
Junyi Li, Tianyi Tang, Gaole He +7
In this paper, we release an open-source library, called TextBox, to provide a unified, modularized, and extensible text generation framework. TextBox aims to support a broad set o…