1 citations · 3 across the 4 of their papers we have counts for
3 papers · 1 filter
On Mitigating Code LLM Hallucinations with API Documentation
Nihal Jain, Robert Kwiatkowski, Baishakhi Ray +2
In this study, we address the issue of API hallucinations in various software engineering contexts. We introduce CloudAPIBench, a new benchmark designed to measure API hallucinatio…
Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies
Junlin Wang, Siddhartha Jain, Dejiao Zhang +3
A diverse array of reasoning strategies has been proposed to elicit the capabilities of large language models. However, in this paper, we point out that traditional evaluations whi…
A Static Evaluation of Code Completion by Large Language Models
Hantian Ding, Varun Kumar, Yuchen Tian +9
Large language models trained on code have shown great potential to increase productivity of software developers. Several execution-based benchmarks have been proposed to evaluate…