12 papers
Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs
Xiaoning Ren, Yinxing Xue, Lei Ma +1
The paper presents Code-MUE, a black‑box method that measures the uncertainty of code‑generating large language models by building execution‑based semantic interaction graphs and c…
Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
Shengming Zhao, Yuchen Shao, Yuheng Huang +4
Retrieval-Augmented Generation (RAG) has emerged as a critical technique for enhancing large language model (LLM) capabilities. However, practitioners face significant challenges w…
Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis
Da Song, Yuheng Huang, Boqi Chen +4
The integration of large language models (LLMs) into autonomous agents has enabled complex tool use, yet in high-stakes domains, these systems must strictly adhere to regulatory st…
Evaluating LLMs on Sequential API Call Through Automated Test Generation
Yuheng Huang, Jiayang Song, Da Song +4
By integrating tools from external APIs, Large Language Models (LLMs) have expanded their promising capabilities in a diverse spectrum of complex real-world tasks. However, testing…
TRUSTVIS: A Multi-Dimensional Trustworthiness Evaluation Framework for Large Language Models
Ruoyu Sun, Da Song, Jiayang Song +2
As Large Language Models (LLMs) continue to revolutionize Natural Language Processing (NLP) applications, critical concerns about their trustworthiness persist, particularly in saf…
AcTracer: Active Testing of Large Language Model via Multi-Stage Sampling
Yuheng Huang, Jiayang Song, Qiang Hu +2
Performance evaluation plays a crucial role in the development life cycle of large language models (LLMs). It estimates the model's capability, elucidates behavior characteristics,…