200 citations · 629 across the 44 of their papers we have counts for
47 papers
Execution-Anchored Hallucination Calibration Reranking for Verilog Code Generation
Guang Yang, Xing Hu, Xiang Chen +2
Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, yet their performance degrades significantly on low-resource Hardware Description Languag…
ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction
Zichun Guo, Yuling Shi, Wenhao Zeng +6
Multimodal Large Language Models (MLLMs) have achieved remarkable performance in Visually Rich Document Understanding (VRDU) tasks, but their capabilities are mainly evaluated on p…
SecCodeBench-V2 Technical Report
Longfei Chen, Ji Zhao, Lanxiao Cui +24
We introduce SecCodeBench-V2, a publicly released benchmark for evaluating Large Language Model (LLM) copilots' capabilities of generating secure code. SecCodeBench-V2 comprises 98…
To Defend Against Cyber Attacks, We Must Teach AI Agents to Hack
Terry Yue Zhuo, Yangruibo Ding, Wenbo Guo +1
For over a decade, cybersecurity has relied on human labor scarcity to limit attackers to high-value targets manually or generic automated attacks at scale. Building sophisticated…
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini +82
AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not…
From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence
Jian Yang, Xianglong Liu, Weifeng Lv +68
Large language models (LLMs) have fundamentally transformed automated software development by enabling direct translation of natural language descriptions into functional code, dri…
Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead
Guang Yang, Wei Zheng, Xiang Chen +14
Code generation has emerged as a critical research area at the intersection of Software Engineering (SE) and Artificial Intelligence (AI), attracting significant attention from bot…
HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities
Xiaoxue Ren, Penghao Jiang, Kaixin Li +6
Web applications are prime targets for cyberattacks as gateways to critical services and sensitive data. Traditional penetration testing is costly and expertise-intensive, making i…
Bamboo: LLM-Driven Discovery of API-Permission Mappings in the Android Framework
Han Hu, Wei Minn, Yonghui Liu +6
The permission mechanism in the Android Framework is integral to safeguarding the privacy of users by managing users' and processes' access to sensitive resources and operations. A…
BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
Terry Yue Zhuo, Xiaolong Jin, Hange Liu +37
Crowdsourced model evaluation platforms, such as Chatbot Arena, enable real-time evaluation from human perspectives to assess the quality of model responses. In the coding domain,…
Training Language Model Agents to Find Vulnerabilities with CTF-Dojo
Terry Yue Zhuo, Dingmin Wang, Hantian Ding +2
Large language models (LLMs) have demonstrated exceptional capabilities when trained within executable runtime environments, notably excelling at software engineering tasks through…
An Empirical Study of Vulnerabilities in Python Packages and Their Detection
Haowei Quan, Junjie Wang, Xinzhe Li +3
In the rapidly evolving software development landscape, Python stands out for its simplicity, versatility, and extensive ecosystem. Python packages, as units of organization, reusa…
The Cream Rises to the Top: Efficient Reranking Method for Verilog Code Generation
Guang Yang, Wei Zheng, Xiang Chen +3
LLMs face significant challenges in Verilog generation due to limited domain-specific knowledge. While sampling techniques improve pass@k metrics, hardware engineers need one trust…
Cyber-Zero: Training Cybersecurity Agents without Runtime
Terry Yue Zhuo, Dingmin Wang, Hantian Ding +2
Large Language Models (LLMs) have achieved remarkable success in software engineering tasks when trained with executable runtime environments, particularly in resolving GitHub issu…
EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
Yuhao Qing, Boyu Zhu, Mingzhe Du +9
Existing code generation benchmarks primarily evaluate functional correctness, with limited focus on code efficiency and often restricted to a single language like Python. To addre…
From Code to Courtroom: LLMs as the New Software Judges
Junda He, Jieke Shi, Terry Yue Zhuo +5
Recently, Large Language Models (LLMs) have been increasingly used to automate SE tasks such as code generation and summarization. However, evaluating the quality of LLM-generated…
CodeArena: A Collective Evaluation Platform for LLM Code Generation
Mingzhe Du, Anh Tuan Luu, Bin Ji +5
Large Language Models (LLMs) have reshaped code generation by synergizing their exceptional comprehension of natural language and programming syntax, thereby substantially boosting…
Identifying and Mitigating API Misuse in Large Language Models
Terry Yue Zhuo, Junda He, Jiamou Sun +4
API misuse in code generated by large language models (LLMs) presents a serious and growing challenge in software development, as although LLMs demonstrate impressive code generati…
GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models
Nizar Islah, Justine Gehring, Diganta Misra +4
The rapid evolution of software libraries presents a significant challenge for code generation models, which must adapt to frequent version updates while maintaining compatibility…
Defending Code Language Models against Backdoor Attacks with Deceptive Cross-Entropy Loss
Guang Yang, Yu Zhou, Xiang Chen +4
Code Language Models (CLMs), particularly those leveraging deep learning, have achieved significant success in code intelligence domain. However, the issue of security, particularl…
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim +30
Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to…
XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts
Yifeng Ding, Jiawei Liu, Yuxiang Wei +2
We introduce XFT, a simple yet powerful training scheme, by simply merging upcycled Mixture-of-Experts (MoE) to unleash the performance limit of instruction-tuned code Large Langua…
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Taishi Nakamura, Mayank Mishra, Simone Tedeschi +42
Pretrained language models are an integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim…
Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Code
Zhou Yang, Zhensu Sun, Terry Zhuo Yue +2
Large language models for code (LLM4Code), which demonstrate strong performance (e.g., high accuracy) in processing source code, have significantly transformed software engineering…