88 citations · 136 across the 34 of their papers we have counts for
7 papers · 1 filter
Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code
Timur Galimzyanov, Sergey Titov, Yaroslav Golubev +1
This paper introduces the human-curated PandasPlotBench dataset, designed to evaluate language models' effectiveness as assistants in visual data exploration. Our benchmark focuses…
One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks
Anastasiia Birillo, Elizaveta Artser, Anna Potriasaeva +6
Students often struggle with solving programming problems when learning to code, especially when they have to do it online, with one of the most common disadvantages of working onl…
Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings
Petr Tsvetkov, Aleksandra Eliseeva, Danny Dig +4
When a Commit Message Generation (CMG) system is integrated into the IDEs and other products at JetBrains, we perform online evaluation based on user acceptance of the generated me…
Long Code Arena: a Set of Benchmarks for Long-Context Code Models
Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov +8
Nowadays, the fields of code and natural language processing are evolving rapidly. In particular, models become better at processing long context windows - supported context sizes…
Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward
Agnia Sergeyuk, Yaroslav Golubev, Timofey Bryksin +1
Context. The last several years saw the emergence of AI assistants for code - multi-purpose AI-based helpers in software engineering. As they become omnipresent in all aspects of s…
Full Line Code Completion: Bringing AI to Desktop
Anton Semenkin, Vitaliy Bibaev, Yaroslav Sokolov +14
In recent years, several industrial solutions for the problem of multi-token code completion appeared, each making a great advance in the area but mostly focusing on cloud-based ru…
Clustering MOOC Programming Solutions to Diversify Their Presentation to Students
Elizaveta Artser, Anastasiia Birillo, Yaroslav Golubev +4
In many MOOCs, whenever a student completes a programming task, they can see previous solutions of other students to find potentially different ways of solving the problem and to l…