4 papers
Prompt Stability in Code LLMs: Measuring Sensitivity across Emotion- and Personality-Driven Variations
Wei Ma, Yixiao Yang, Jingquan Ge +2
Code generation models are widely used in software development, yet their sensitivity to prompt phrasing remains under-examined. Identical requirements expressed with different emo…
VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities
Weizhe Wang, Wei Ma, Qiang Hu +6
The adoption of Large Language Models (LLMs) for automated software vulnerability patching has shown promising outcomes on carefully curated evaluation sets. Nevertheless, existing…
Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol
Wei Ma, Yixiao Yang, Qiang Hu +8
Applications of Large Language Models~(LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and mult…
Importance Sampling is All You Need: Predict LLM's performance on new benchmark by reusing existing benchmark
Junjie Shi, Wei Ma, Shi Ying +3
With the rapid advancement of large language models , code generation has become a key benchmark for evaluating LLM capabilities. However, existing benchmarks face two major challe…