6 citations · 19 across the 21 of their papers we have counts for
24 papers
LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation
Han Li, Zhemin Fang, Rili Feng +8
Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development. Existing ben…
Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment
Kirill Vasilevski, Ximing Dong, Benjamin Rombaut +8
LLMs have substantially improved software engineering yet real-world development requires architectural understanding. Such understanding is prohibitively expensive to label manual…
When Elo Lies: Hidden Biases in Codeforces-Based Evaluation of Large Language Models
Shenyu Zheng, Ximing Dong, Xiaoshuang Liu +6
As Large Language Models (LLMs) achieve breakthroughs in complex reasoning, Codeforces-based Elo ratings have emerged as a prominent metric for evaluating competitive programming c…
Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
Ximing Dong, Shaowei Wang, Dayi Lin +2
Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The issue is exacerbated in Large R…
SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints
Zhiyu Fan, Kirill Vasilevski, Dayi Lin +6
The advancement of large language models (LLMs) and code agents has demonstrated significant potential to assist software engineering (SWE) tasks, such as autonomous issue resoluti…
RepoForge: Training a SOTA Fast-thinking SWE Agent with an End-to-End Data Curation Pipeline Synergizing SFT and RL at Scale
Zhilong Chen, Chengzong Zhao, Boyuan Chen +9
Training software engineering (SWE) LLMs is bottlenecked by expensive infrastructure, inefficient evaluation pipelines, scarce training data, and costly quality control. We present…
Towards Reliable Generation of Executable Workflows by Foundation Models
Sogol Masoumzadeh, Keheliya Gallaba, Dayi Lin +1
Recent advancements in Foundation Models (FMs) have demonstrated significant progress in processing complex natural language to perform intricate tasks. Successfully executing thes…
Agentic Software Engineering: Foundational Pillars and a Research Roadmap
Ahmed E. Hassan, Hao Li, Dayi Lin +4
Agentic Software Engineering (SE 3.0) represents a new era where intelligent agents are tasked not with simple code generation, but with achieving complex, goal-oriented SE objecti…
SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation
Gustavo A. Oliva, Gopi Krishnan Rajbahadur, Aaditya Bhatia +7
High-quality labeled datasets are crucial for training and evaluating foundation models in software engineering, but creating them is often prohibitively expensive and labor-intens…
The Hitchhikers Guide to Production-ready Trustworthy Foundation Model powered Software (FMware)
Kirill Vasilevski, Benjamin Rombaut, Gopi Krishnan Rajbahadur +10
Foundation Models (FMs) such as Large Language Models (LLMs) are reshaping the software industry by enabling FMware, systems that integrate these FMs as core components. In this KD…
Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement
Keheliya Gallaba, Ali Arabat, Dayi Lin +2
Foundation Models (FMs) have shown remarkable capabilities in various natural language tasks. However, their ability to accurately capture stakeholder requirements remains a signif…
Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization
Ximing Dong, Shaowei Wang, Dayi Lin +1
Optimizing Large Language Model (LLM) performance requires well-crafted prompts, but manual prompt engineering is labor-intensive and often ineffective. Automated prompt optimizati…
Engineering AI Judge Systems
Jiahuei Lin, Dayi Lin, Sky Zhang +1
AI judge systems are designed to automatically evaluate Foundation Model-powered software (i.e., FMware). Due to the intrinsic dynamic and stochastic nature of FMware, the developm…
Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation Models
Kirill Vasilevski, Dayi Lin, Ahmed E. Hassan
To balance the quality and inference cost of a Foundation Model (FM, such as large language models (LLMs)) powered software, people often opt to train a routing model that routes r…
Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agents
Benjamin Rombaut, Sogol Masoumzadeh, Kirill Vasilevski +2
Large language models (LLMs) are increasingly integrated into autonomous systems, giving rise to a new class of software known as Agentware, where LLM-powered agents perform comple…
PromptExp: Multi-granularity Prompt Explanation of Large Language Models
Ximing Dong, Shaowei Wang, Dayi Lin +4
Large Language Models excel in tasks like natural language understanding and text generation. Prompt engineering plays a critical role in leveraging LLM effectively. However, LLMs…
From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap
Gopi Krishnan Rajbahadur, Gustavo A. Oliva, Dayi Lin +2
The rapid expansion of foundation models (FMs), such as large language models (LLMs), has given rise to FMware, software systems that integrate FM(s) as core components. While buil…
Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap
Ahmed E. Hassan, Gustavo A. Oliva, Dayi Lin +3
The rise of AI-assisted software engineering (SE 2.0), powered by Foundation Models (FMs) and FM-powered coding assistants, has shown promise in improving developer productivity. H…
Data Quality Antipatterns for Software Analytics
Aaditya Bhatia, Dayi Lin, Gopi Krishnan Rajbahadur +2
Background: Data quality is vital in software analytics, particularly for machine learning (ML) applications like software defect prediction (SDP). Despite the widespread use of ML…
SimClone: Detecting Tabular Data Clones using Value Similarity
Xu Yang, Gopi Krishnan Rajbahadur, Dayi Lin +3
Data clones are defined as multiple copies of the same data among datasets. Presence of data clones between datasets can cause issues such as difficulties in managing data assets a…
Rethinking Software Engineering in the Foundation Model Era: From Task-Driven AI Copilots to Goal-Driven AI Pair Programmers
Ahmed E. Hassan, Gustavo A. Oliva, Dayi Lin +3
The advent of Foundation Models (FMs) and AI-powered copilots has transformed the landscape of software development, offering unprecedented code completion capabilities and enhanci…
A Framework for Real-time Safeguarding the Text Generation of Large Language Model
Ximing Dong, Dayi Lin, Shaowei Wang +1
Large Language Models (LLMs) have significantly advanced natural language processing (NLP) tasks but also pose ethical and societal risks due to their propensity to generate harmfu…
Rethinking Software Engineering in the Foundation Model Era: A Curated Catalogue of Challenges in the Development of Trustworthy FMware
Ahmed E. Hassan, Dayi Lin, Gopi Krishnan Rajbahadur +9
Foundation models (FMs), such as Large Language Models (LLMs), have revolutionized software development by enabling new use cases and business models. We refer to software built us…
Can I use this publicly available dataset to build commercial AI software? -- A Case Study on Publicly Available Image Datasets
Gopi Krishnan Rajbahadur, Erika Tuck, Li Zi +5
Publicly available datasets are one of the key drivers for commercial AI software. The use of publicly available datasets is governed by dataset licenses. These dataset licenses ou…