11 papers
Last Secure Code Benchmark: Is the Code You Vibe Secure?
Tianyi Wu, Chao Ge, Peng Chen +12
AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark, a comprehensive evaluation of…
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Lin Shi, Haowei Lin, Zixuan Zhu +123
Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a…
SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
Yuling Shi, Jinghan Xu, Kelin Fu +12
As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under se…
Knowledge-Enhanced Agentic Vulnerability Repair
Sicong Cao, Hao Ma, Le Yu +8
Frontier foundation models have changed the math on vulnerability discovery, but the bigger challenge is how the remediation side keeps up. Despite recent progresses in Automated V…
PrivCode++: Latent-Conditioned Differentially Private Code Generation for Comprehensive Guarantees
Zheng Liu, Chen Gong, Terry Yue Zhuo +6
Large language models fine-tuned on instruction-code pairs may memorize and subsequently leak sensitive training data. Existing differentially private (DP) code generation methods…
SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
Man Ho Lam, Chaozheng Wang, Hange Liu +5
Coding agents powered by large language models are increasingly expected to perform realistic software maintenance tasks beyond isolated issue resolution. Existing benchmarks have…
Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code
Myeongsoo Kim, Dingmin Wang, Siwei Cui +6
Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed withou…
Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Model
Tianyi Wu, Mingzhe Du, Yue Liu +4
Large language models (LLMs) are increasingly used in software development, yet their tendency to generate insecure code remains a major barrier to real-world deployment. Existing…
Watermarking LLM Agent Trajectories
Wenlong Meng, Chen Gong, Terry Yue Zhuo +6
LLM agents rely heavily on high-quality trajectory data to guide their problem-solving behaviors, yet producing such data requires substantial task design, high-capacity model gene…
PrivCode: When Code Generation Meets Differential Privacy
Zheng Liu, Chen Gong, Terry Yue Zhuo +4
Large language models (LLMs) have presented outstanding performance in code generation and completion. However, fine-tuning these models on private datasets can raise privacy and p…
A Mixture of Linear Corrections Generates Secure Code
Weichen Yu, Ravi Mangal, Terry Zhuo +2
Large language models (LLMs) have become proficient at sophisticated code-generation tasks, yet remain ineffective at reliably detecting or avoiding code vulnerabilities. Does this…