88 citations · 136 across the 34 of their papers we have counts for
39 papers
Coding-Agent Benchmarks Should Match Their Users' Task Flows
Igor Slinko, Yaroslav Golubev, Sergey Titov
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which…
Challenge on Optimization of Context Collection for Code Completion
Dmitry Ustalov, Egor Bogomolov, Alexander Bezzubov +4
The rapid advancement of workflows and methods for software engineering using AI emphasizes the need for a systematic evaluation and analysis of their ability to leverage informati…
Observing Fine-Grained Changes in Jupyter Notebooks During Development Time
Sergey Titov, Konstantin Grotov, Cristina Sarasua +5
In software engineering research, the analysis of fine-grained logs led to significant innovations in areas such as refactoring, security, and code completion. However, even though…
Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code
Timur Galimzyanov, Sergey Titov, Yaroslav Golubev +1
This paper introduces the human-curated PandasPlotBench dataset, designed to evaluate language models' effectiveness as assistants in visual data exploration. Our benchmark focuses…
One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks
Anastasiia Birillo, Elizaveta Artser, Anna Potriasaeva +6
Students often struggle with solving programming problems when learning to code, especially when they have to do it online, with one of the most common disadvantages of working onl…
Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings
Petr Tsvetkov, Aleksandra Eliseeva, Danny Dig +4
When a Commit Message Generation (CMG) system is integrated into the IDEs and other products at JetBrains, we perform online evaluation based on user acceptance of the generated me…
Long Code Arena: a Set of Benchmarks for Long-Context Code Models
Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov +8
Nowadays, the fields of code and natural language processing are evolving rapidly. In particular, models become better at processing long context windows - supported context sizes…
Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward
Agnia Sergeyuk, Yaroslav Golubev, Timofey Bryksin +1
Context. The last several years saw the emergence of AI assistants for code - multi-purpose AI-based helpers in software engineering. As they become omnipresent in all aspects of s…
Full Line Code Completion: Bringing AI to Desktop
Anton Semenkin, Vitaliy Bibaev, Yaroslav Sokolov +14
In recent years, several industrial solutions for the problem of multi-token code completion appeared, each making a great advance in the area but mostly focusing on cloud-based ru…
Clustering MOOC Programming Solutions to Diversify Their Presentation to Students
Elizaveta Artser, Anastasiia Birillo, Yaroslav Golubev +4
In many MOOCs, whenever a student completes a programming task, they can see previous solutions of other students to find potentially different ways of solving the problem and to l…
From Commit Message Generation to History-Aware Commit Message Completion
Aleksandra Eliseeva, Yaroslav Sokolov, Egor Bogomolov +3
Commit messages are crucial to software development, allowing developers to track changes and collaborate effectively. Despite their utility, most commit messages lack important in…
Overcoming the Mental Set Effect in Programming Problem Solving
Agnia Sergeyuk, Sergey Titov, Yaroslav Golubev +1
This paper adopts a cognitive psychology perspective to investigate the recurring mistakes in code resulting from the mental set (Einstellung) effect. The Einstellung effect is the…
Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses
Anastasiia Birillo, Elizaveta Artser, Yaroslav Golubev +4
In this work, we developed an algorithm for detecting code quality issues in the templates of online programming tasks, validated it, and conducted an empirical study on the datase…
Optimizing Duplicate Size Thresholds in IDEs
Konstantin Grotov, Sergey Titov, Alexandr Suhinin +2
In this paper, we present an approach for transferring an optimal lower size threshold for clone detection from one language to another by analyzing their clone distributions. We s…
Just-in-Time Code Duplicates Extraction
Eman Abdullah AlOmar, Anton Ivanov, Zarina Kurbatova +7
Refactoring is a critical task in software maintenance, and is usually performed to enforce better design and coding practices, while coping with design defects. The Extract Method…
Analyzing the Quality of Submissions in Online Programming Courses
Maria Tigina, Anastasiia Birillo, Yaroslav Golubev +3
Programming education should aim to provide students with a broad range of skills that they will later use while developing software. An important aspect in this is their ability t…
Predicting Tags For Programming Tasks by Combining Textual And Source Code Data
Artyom Lobanov, Egor Bogomolov, Yaroslav Golubev +2
Competitive programming remains a very popular activity that combines both software engineering and education. In order to prepare and to practice, contestants use extensive archiv…
So Much in So Little: Creating Lightweight Embeddings of Python Libraries
Yaroslav Golubev, Egor Bogomolov, Egor Bulychev +1
In software engineering, different approaches and machine learning models leverage different types of data: source code, textual information, historical data. An important part of…
All You Need Is Logs: Improving Code Completion by Learning from Anonymous IDE Usage Logs
Vitaliy Bibaev, Alexey Kalina, Vadim Lomshakov +4
In this work, we propose an approach for collecting completion usage logs from the users in an IDE and using them to train a machine learning based model for ranking completion can…
Aggregation of Stack Trace Similarities for Crash Report Deduplication
Nikolay Karasov, Aleksandr Khvorov, Roman Vasiliev +2
The automatic collection of stack traces in bug tracking systems is an integral part of many software projects and their maintenance. However, such reports often contain a lot of d…
A Large-Scale Comparison of Python Code in Jupyter Notebooks and Scripts
Konstantin Grotov, Sergey Titov, Vladimir Sotnikov +2
In recent years, Jupyter notebooks have grown in popularity in several domains of software engineering, such as data science, machine learning, and computer science education. Thei…
Lupa: A Framework for Large Scale Analysis of the Programming Language Usage
Anna Vlasova, Maria Tigina, Ilya Vlasov +3
In this paper, we present Lupa - a framework for large-scale analysis of the programming language usage. Lupa is a command line tool that uses the power of the IntelliJ Platform un…
DapStep: Deep Assignee Prediction for Stack Trace Error rePresentation
Denis Sushentsev, Aleksandr Khvorov, Roman Vasiliev +2
The task of finding the best developer to fix a bug is called bug triage. Most of the existing approaches consider the bug triage task as a classification problem, however, classif…
ReSplit: Improving the Structure of Jupyter Notebooks by Re-Splitting Their Cells
Sergey Titov, Yaroslav Golubev, Timofey Bryksin
Jupyter notebooks represent a unique format for programming - a combination of code and Markdown with rich formatting, separated into individual cells. We propose to perceive a Jup…