5 papers · 1 filter
What Survives the Next Model? Benchmarking LLM-Based Techniques Against Single-Prompts
Nahian Salsabil, Joy Saha, Simantika Bhattacharjee Dristi +4
The software engineering research community has enthusiastically embraced the integration of Large Language Models (LLMs) into complex techniques to solve a wide variety of tasks.…
SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References
Simantika Bhattacharjee Dristi, Matthew B. Dwyer
Source code summaries have traditionally been evaluated from a human developer's perspective, with quality determined by how closely they resemble developer-written references and…
A Differential Fuzzing-Based Evaluation of Functional Equivalence in LLM-Generated Code Refactorings
Simantika Bhattacharjee Dristi, Matthew B. Dwyer
With the rapid adoption of large language models (LLMs) in automated code refactoring, assessing and ensuring functional equivalence between LLM-generated refactoring and the origi…
Analyzing and Mitigating Surface Bias in Code Evaluation Metrics
Simantika Bhattacharjee Dristi, Matthew B. Dwyer
With the increasing popularity of large language models (LLMs) and LLM-based agents, reliable and effective code evaluation metrics (CEMs) have become crucial for progress across s…
The Fault in our Stars: Quality Assessment of Code Generation Benchmarks
Mohammed Latif Siddiq, Simantika Dristi, Joy Saha +1
Large Language Models (LLMs) are gaining popularity among software engineers. A crucial aspect of developing effective code generation LLMs is to evaluate these models using a robu…