Caiqi Zhang

Code Is More Than Text: Uncertainty Estimation for Code Generation

Yuling Shi, Caiqi Zhang, Yuexian Li +4

Large language models (LLMs) are increasingly deployed as code generators, where silently wrong programs pose real safety and reliability risks. Reliable uncertainty estimation (UE…

When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs

Zhichao Yang, Caiqi Zhang, Ruihan Yang +3

Calibration evaluates whether a model confidence aligns with its empirical accuracy. Existing studies often compare the calibration of different large language models using global…

Value of Information: A Framework for Human-Agent Communication

Yijiang River Dong, Tiancheng Hu, Zheng Hui +4

Large Language Model (LLM) agents deployed for real-world tasks face a fundamental dilemma: user requests are underspecified, yet agents must decide whether to act on incomplete in…

Agent-as-a-Judge

Runyang You, Hongru Cai, Caiqi Zhang +5

LLM-as-a-Judge has revolutionized AI evaluation by leveraging large language models for scalable assessments. However, as evaluands become increasingly complex, specialized, and mu…

Demystifying Multi-Agent Debate: The Role of Confidence and Diversity

Xiaochen Zhu, Caiqi Zhang, Yizhou Chi +3

Multi-agent debate (MAD) is widely used to improve large language model (LLM) performance through test-time scaling, yet recent work shows that vanilla MAD often underperforms simp…

Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective

Beiduo Chen, Tiancheng Hu, Caiqi Zhang +3

Reasoning-tuned LLMs utilizing long Chain-of-Thought (CoT) excel at single-answer tasks, yet their ability to model Human Label Variation--which requires capturing probabilistic am…

Confidence Estimation for LLMs in Multi-turn Interactions

Caiqi Zhang, Ruihan Yang, Xiaochen Zhu +5

While confidence estimation is a promising direction for mitigating hallucinations in Large Language Models (LLMs), current research overwhelmingly focuses on single-turn settings.…

Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models

Ej Zhou, Caiqi Zhang, Tiancheng Hu +4

Confidence calibration, the alignment of a model's predicted confidence with its actual accuracy, is crucial for the reliable deployment of Large Language Models (LLMs). However, t…

All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning

Caiqi Zhang, Chang Shu, Ehsan Shareghi +1

Confidence estimation is essential for the reliable deployment of large language models (LLMs). Existing methods are primarily designed for factual QA tasks and often fail to gener…

Multi-Trigger Poisoning Amplifies Backdoor Vulnerabilities in LLMs

Sanhanat Sivapiromrat, Caiqi Zhang, Marco Basaldella +1

Recent studies have shown that Large Language Models (LLMs) are vulnerable to data poisoning attacks, where malicious training examples embed hidden behaviours triggered by specifi…

LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generations

Caiqi Zhang, Xiaochen Zhu, Chengzu Li +2

Hallucination remains a major challenge for the safe and trustworthy deployment of large language models (LLMs) in factual content generation. Prior work has explored confidence es…

UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generation

Ruihan Yang, Caiqi Zhang, Zhisong Zhang +4

Large Language Models (LLMs) are prone to hallucination, particularly in long-form generations. A promising direction to mitigate hallucination is to teach LLMs to express uncertai…

LoGU: Long-form Generation with Uncertainty Expressions

Ruihan Yang, Caiqi Zhang, Zhisong Zhang +5

While Large Language Models (LLMs) demonstrate impressive capabilities, they still struggle with generating factually incorrect content (i.e., hallucinations). A promising approach…

Atomic Calibration of LLMs in Long-Form Generations

Caiqi Zhang, Ruihan Yang, Zhisong Zhang +4

Large language models (LLMs) often suffer from hallucinations, posing significant challenges for real-world applications. Confidence calibration, as an effective indicator of hallu…

Conformity in Large Language Models

Xiaochen Zhu, Caiqi Zhang, Tom Stafford +2

The conformity effect describes the tendency of individuals to align their responses with the majority. Studying this bias in large language models (LLMs) is crucial, as LLMs are i…

TopViewRS: Vision-Language Models as Top-View Spatial Reasoners

Chengzu Li, Caiqi Zhang, Han Zhou +3

Top-view perspective denotes a typical way in which humans read and reason over different types of maps, and it is vital for localization and navigation of humans as well as of `no…

Can Large Language Models Generate High-quality Patent Claims?

Lekang Jiang, Caiqi Zhang, Pascal A Scherz +1

Large language models (LLMs) have shown exceptional performance across various text generation tasks but remain under-explored in the patent domain, which offers highly structured…

LUQ: Long-text Uncertainty Quantification for LLMs

Caiqi Zhang, Fangyu Liu, Marco Basaldella +1

Large Language Models (LLMs) have demonstrated remarkable capability in a variety of NLP tasks. However, LLMs are also prone to generate nonfactual content. Uncertainty Quantificat…