12 citations · 19 across the 7 of their papers we have counts for
7 papers
MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks
Artem Chervyakov, Alexander Kharitonov, Pavel Zadorozhny +20
Advancements in LLMs have enhanced task automation in software engineering; however, current evaluations primarily focus on natural language tasks, overlooking code quality. Most b…
SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
Pavel Adamenko, Mikhail Ivanov, Aidar Valeev +6
The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench datas…
YABLoCo: Yet Another Benchmark for Long Context Code Generation
Aidar Valeev, Roman Garaev, Vadim Lomshakov +3
Large Language Models demonstrate the ability to solve various programming tasks, including code generation. Typically, the performance of LLMs is measured on benchmarks with small…
Leveraging Large Language Models in Code Question Answering: Baselines and Issues
Georgy Andryushchenko, Vladimir Ivanov, Vladimir Makharev +2
Question answering over source code provides software engineers and project managers with helpful information about the implemented features of a software product. This paper prese…
Experiments with LVT and FRE for Transformer model
Ilshat Gibadullin, Aidar Valeev
In this paper, we experiment with Large Vocabulary Trick and Feature-rich encoding applied to the Transformer model for Text Summarization. We could not achieve better results, tha…
A Survey of Methods to Leverage Monolingual Data in Low-resource Neural Machine Translation
Ilshat Gibadullin, Aidar Valeev, Albina Khusainova +1
Neural machine translation has become the state-of-the-art for language pairs with large parallel corpora. However, the quality of machine translation for low-resource languages le…