15 papers
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12
Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In…
Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation
Qiming Zhu, Jialun Cao, Xuanang Chen +6
Current research on large language models (LLMs) with retrieval-augmented code generation (RACG) has largely focused on single-language settings, leaving their cross-lingual effect…
MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing
Congying Xu, Songqiang Chen, Jiarong Wu +4
While a recent study reveals that many developer-written test cases can encode a reusable Metamorphic Relation (MR), over 70% of them directly hard-code the source input and follow…
MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases
Congying Xu, Valerio Terragni, Hengcheng Zhu +2
Metamorphic Testing (MT) alleviates the oracle problem by defining oracles based on metamorphic relations (MRs), that govern multiple related inputs and their outputs. However, des…
ModelWisdom: An Integrated Toolkit for TLA+ Model Visualization, Digest and Repair
Zhiyong Chen, Jialun Cao, Chang Xu +1
Model checking in TLA+ provides strong correctness guarantees, yet practitioners continue to face significant challenges in interpreting counterexamples, understanding large state-…
EmbedAgent: Benchmarking Large Language Models in Embedded System Development
Ruiyang Xu, Jialun Cao, Mingyuan Wu +6
Large Language Models (LLMs) have shown promise in various tasks, yet few benchmarks assess their capabilities in embedded system development. In this paper, we introduce EmbedAgen…