42 citations · 160 across the 27 of their papers we have counts for
29 papers
GraphDroid: Asynchronous LLM-Based Mobile App GUI Testing via History-Aware Exploration and Hybrid Intent Fulfillment
Xiaolei Li, Jialun Cao, Zhijian Hou +3
Automated GUI testing is a widely adopted technique for ensuring mobile application quality by simulating user interactions to exercise functionalities. Despite the research breakt…
SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
Yuling Shi, Jinghan Xu, Kelin Fu +12
As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under se…
Inside the Skill Market: From Software Engineering Activities to Reusable Agent Skills
Jialun Cao, Xinru Yan, Songqiang Chen +3
Software engineering (abbrev. SE) has continuously evolved through increasingly powerful forms of reuse, from source code and libraries to components and services. Recent advances…
Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training
Yuanyi Wang, Yifan Yang, Su Lu +9
Continual post-training aims to extend large language models (LLMs) with new knowledge, skills, and behaviors, yet it remains unclear when sequential updates enable capability tran…
Can Large Language Models Model Programs Formally?
Zhiyong Chen, Jialun Cao, Jiarong Wu +2
In the digital age, ensuring the correctness, safety, and reliability of software through formal verification is paramount, particularly as software increasingly underpins critical…
iCoRe: An Iterative Correlation-Aware Retriever for Bug Reproduction Test Generation
Junyi Wang, Jialun Cao, Zhongxin Liu
Automatically generating bug reproduction tests (BRT) from issue descriptions is crucial for software maintenance. LLM-based approaches have shown great potential for this task. Th…
SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark
Boxi Yu, Yang Cao, Yuzhong Zhang +9
The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we show that this performance is inflated. Our re-evaluation reveals th…
ModelWisdom: An Integrated Toolkit for TLA+ Model Visualization, Digest and Repair
Zhiyong Chen, Jialun Cao, Chang Xu +1
Model checking in TLA+ provides strong correctness guarantees, yet practitioners continue to face significant challenges in interpreting counterexamples, understanding large state-…
Can Emulating Semantic Translation Help LLMs with Code Translation? A Study Based on Pseudocode
Songqiang Chen, Congying Xu, Jingyi Chen +3
Although large language models (LLMs) show promising potential in code translation, they still struggle to generate accurate translations using the commonly adopted direct code-to-…
What Makes In-Context Examples Effective for Code Generation?
Dongze Li, Songqiang Chen, Jialun Cao +1
In-Context Learning (ICL) has emerged as a promising solution to enhance the code generation capabilities of Large Language Models (LLMs) by incorporating code examples inside the…
EmbedAgent: Benchmarking Large Language Models in Embedded System Development
Ruiyang Xu, Jialun Cao, Mingyuan Wu +6
Large Language Models (LLMs) have shown promise in various tasks, yet few benchmarks assess their capabilities in embedded system development. In this paper, we introduce EmbedAgen…
Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation
Qiming Zhu, Jialun Cao, Xuanang Chen +6
Current research on large language models (LLMs) with retrieval-augmented code generation (RACG) has largely focused on single-language settings, leaving their cross-lingual effect…
ReuseDroid: A VLM-empowered Android UI Test Migrator Boosted by Active Feedback
Xiaolei Li, Jialun Cao, Yepang Liu +2
GUI testing is an essential quality assurance process in mobile app development. However, the creation and maintenance of GUI tests for mobile apps are resource-intensive and costl…
When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?
Jingyi Chen, Songqiang Chen, Jialun Cao +2
Retrieval-augmented generation (RAG) has increasingly shown its power in extending large language models' (LLMs') capability beyond their pre-trained knowledge. Existing works have…
Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval
Jiarong Wu, Songqiang Chen, Jialun Cao +2
Existing code generation benchmarks for Large Language Models (LLMs) such as HumanEval and MBPP are designed to study LLMs' end-to-end performance, where the benchmarks feed a prob…
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
Jialun Cao, Yuk-Kit Chan, Zixuan Ling +12
Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In…
CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkit
Jialun Cao, Songqiang Chen, Wuqi Zhang +2
Data contamination presents a critical barrier preventing widespread industrial adoption of advanced software engineering techniques that leverage code language models (CLMs). This…
DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation
Qiming Zhu, Jialun Cao, Yaojie Lu +4
Code benchmarks such as HumanEval are widely adopted to evaluate the capabilities of Large Language Models (LLMs), providing insights into their strengths and weaknesses. However,…
MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing
Congying Xu, Songqiang Chen, Jiarong Wu +4
While a recent study reveals that many developer-written test cases can encode a reusable Metamorphic Relation (MR), over 70% of them directly hard-code the source input and follow…
CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution
Ruiyang Xu, Jialun Cao, Yaojie Lu +6
Code benchmarks such as HumanEval are widely adopted to evaluate Large Language Models' (LLMs) coding capabilities. However, there is an unignorable programming language bias in ex…
JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models
Jialun Cao, Zhiyong Chen, Jiarong Wu +2
Code generation benchmarks such as HumanEval are widely adopted to evaluate LLMs' capabilities. However, after consolidating the latest 24 benchmarks, we noticed three significant…
Enhancing Differential Testing With LLMs For Testing Deep Learning Libraries
Meiziniu Li, Dongze Li, Jianmeng Liu +3
Differential testing offers a promising strategy to alleviate the test oracle problem by comparing the test results between alternative implementations. However, existing different…
Enchanting Program Specification Synthesis by Large Language Models using Static Analysis and Program Verification
Cheng Wen, Jialun Cao, Jie Su +6
Formal verification provides a rigorous and systematic approach to ensure the correctness and reliability of software systems. Yet, constructing specifications for the full proof r…
Concerned with Data Contamination? Assessing Countermeasures in Code Language Model
Jialun Cao, Wuqi Zhang, Shing-Chi Cheung
Various techniques have been proposed to leverage the capabilities of code language models (CLMs) for SE tasks. While these techniques typically evaluate their effectiveness using…