115 citations · 116 across the 4 of their papers we have counts for
4 papers
Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Required
Egor Shibaev, Vera Kudrevskaia, Timur Galimzyanov +9
Post-training papers, model cards, and blog posts often treat scores on a small set of coding benchmarks (e.g., SWE-bench and LiveCodeBench) as evidence of broad coding capability,…
Kotlin ML Pack: Technical Report
Sergey Titov, Mikhail Evtikhiev, Anton Shapkin +7
In this technical report, we present three novel datasets of Kotlin code: KStack, KStack-clean, and KExercises. We also describe the results of fine-tuning CodeLlama and DeepSeek m…
Judging Adam: Studying the Performance of Optimization Methods on ML4SE Tasks
Dmitry Pasechnyuk, Anton Prazdnichnykh, Mikhail Evtikhiev +1
Solving a problem with a deep learning model requires researchers to optimize the loss function with a certain optimization method. The research community has developed more than a…
Out of the BLEU: how should we assess quality of the Code Generation models?
Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov +1
In recent years, researchers have created and introduced a significant number of various code generation models. As human evaluation of every new model version is unfeasible, the c…