5 papers
SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
Pavel Adamenko, Mikhail Ivanov, Aidar Valeev +6
The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench datas…
Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Maria Ivanova, Pavel Zadorozhny, Rodion Levichev +5
LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problem…
MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks
Artem Chervyakov, Alexander Kharitonov, Pavel Zadorozhny +20
Advancements in LLMs have enhanced task automation in software engineering; however, current evaluations primarily focus on natural language tasks, overlooking code quality. Most b…
Exploring the Potential of Conversational Test Suite Based Program Repair on SWE-bench
Anton Cheshkov, Pavel Zadorozhny, Rodion Levichev +2
Automatic program repair at project level may open yet to be seen opportunities in various fields of human activity. Since the SWE-Bench challenge was presented, we have seen numer…
Finetuning Large Language Models for Vulnerability Detection
Alexey Shestov, Rodion Levichev, Ravil Mussabayev +3
This paper presents the results of finetuning large language models (LLMs) for the task of detecting vulnerabilities in source code. We leverage WizardCoder, a recent improvement o…