29 citations · 152 across the 38 of their papers we have counts for
40 papers
MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding
Bosi Wen, Cunxiang Wang, Jiayi Gui +6
Recently, the rapid development of large language models (LLMs) has reshaped software engineering by enabling autonomous code agents that plan, execute, and utilize external tools…
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
Bosi Wen, Yilin Niu, Cunxiang Wang +5
Instruction-following is a foundational capability of large language models (LLMs), with its improvement hinging on scalable and accurate feedback from judge models. However, the r…
Survive at All Costs: Exploring LLM's Risky Behaviors under Survival Pressure
Yida Lu, Jianwei Fang, Xuyang Shao +7
As Large Language Models (LLMs) evolve from chatbots to agentic assistants, they are increasingly observed to exhibit risky behaviors when subjected to survival pressure, such as t…
IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
Bosi Wen, Yilin Niu, Cunxiang Wang +6
Instruction-following is a fundamental ability of Large Language Models (LLMs), requiring their generated outputs to follow multiple constraints imposed in input instructions. Nume…
Training Language Model to Critique for Better Refinement
Tianshu Yu, Chao Xiang, Mingchuan Yang +8
Large language models (LLMs) have demonstrated remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. However, limit…
HPSS: Heuristic Prompting Strategy Search for LLM Evaluators
Bosi Wen, Pei Ke, Yufei Sun +6
Since the adoption of large language models (LLMs) for text evaluation has become increasingly prevalent in the field of natural language processing (NLP), a series of existing wor…
The Superalignment of Superhuman Intelligence with Large Language Models
Minlie Huang, Yingkang Wang, Shiyao Cui +2
We have witnessed superhuman intelligence thanks to the fast development of large language models and multimodal language models. As the application of such superhuman models becom…
CharacterBench: Benchmarking Character Customization of Large Language Models
Jinfeng Zhou, Yongkang Huang, Bosi Wen +13
Character-based dialogue (aka role-playing) enables users to freely customize characters for interaction, which often relies on LLMs, raising the need to evaluate LLMs' character c…
Benchmarking Complex Instruction-Following with Multiple Constraints Composition
Bosi Wen, Pei Ke, Xiaotao Gu +11
Instruction following is one of the fundamental capabilities of large language models (LLMs). As the ability of LLMs is constantly improving, they have been increasingly applied to…
From Theft to Bomb-Making: The Ripple Effect of Unlearning in Defending Against Jailbreak Attacks
Zhexin Zhang, Junxiao Yang, Yida Lu +5
Large Language Models (LLMs) are known to be vulnerable to jailbreak attacks. An important observation is that, while different types of jailbreak attacks can generate significantl…
AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Models
Jiale Cheng, Yida Lu, Xiaotao Gu +6
Although Large Language Models (LLMs) are becoming increasingly powerful, they still exhibit significant but subtle weaknesses, such as mistakes in instruction-following or coding…
Learning Task Decomposition to Assist Humans in Competitive Programming
Jiaxin Wen, Ruiqi Zhong, Pei Ke +3
When using language models (LMs) to solve complex problems, humans might struggle to understand the LM-generated solutions and repair the flawed ones. To assist humans in repairing…
Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering
Zhihua Wen, Zhiliang Tian, Zexin Jian +5
Large Language Models (LLMs) are widely used for knowledge-seeking yet suffer from hallucinations. The knowledge boundary (KB) of an LLM limits its factual understanding, beyond wh…
ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectors
Zhexin Zhang, Yida Lu, Jingyuan Ma +8
The safety of Large Language Models (LLMs) has gained increasing attention in recent years, but there still lacks a comprehensive approach for detecting safety issues within LLMs'…
Towards Efficient Exact Optimization of Language Model Alignment
Haozhe Ji, Cheng Lu, Yilin Niu +5
The alignment of language models with human preferences is vital for their application in real-world tasks. The problem is formulated as optimizing the model's policy to maximize t…
Unveiling the Implicit Toxicity in Large Language Models
Jiaxin Wen, Pei Ke, Hao Sun +4
The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. While recent st…
AlignBench: Benchmarking Chinese Alignment of Large Language Models
Xiao Liu, Xuanyu Lei, Shengyuan Wang +15
Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, the effective evaluation of alignment for emerging Ch…
CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation
Pei Ke, Bosi Wen, Zhuoer Feng +9
Since the natural language processing (NLP) community started to make large language models (LLMs) act as a critic to evaluate the quality of generated texts, most of the existing…
Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization
Zhexin Zhang, Junxiao Yang, Pei Ke +3
While significant attention has been dedicated to exploiting weaknesses in LLMs through jailbreaking attacks, there remains a paucity of effort in defending against these attacks.…
Black-Box Prompt Optimization: Aligning Large Language Models without Model Training
Jiale Cheng, Xiao Liu, Kehan Zheng +5
Large language models (LLMs) have shown impressive success in various applications. However, these models are often not well aligned with human intents, which calls for additional…
Language Model Decoding as Direct Metrics Optimization
Haozhe Ji, Pei Ke, Hongning Wang +1
Despite the remarkable advances in language modeling, current mainstream decoding methods still struggle to generate texts that align with human texts across different aspects. In…
DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering
Pei Ke, Fei Huang, Fei Mi +4
Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. Specifically, most of the well-performed…
Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning
Chujie Zheng, Pei Ke, Zheng Zhang +1
It has always been an important yet challenging problem to control language models to avoid generating texts with undesirable attributes, such as toxic language and unnatural repet…
Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation
Fei Huang, Pei Ke, Minlie Huang
Non-AutoRegressive (NAR) text generation models have drawn much attention because of their significantly faster decoding speed and good generation quality in machine translation. H…