11 papers
SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
Pavel Adamenko, Mikhail Ivanov, Aidar Valeev +6
The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench datas…
Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
Ivan Sedykh, Nikita Sorokin, Valentin Malykh
Recent advances in masked diffusion language models (MDLMs) narrow the quality gap to autoregressive LMs, but their sampling remains expensive because generation requires many full…
ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
Dmitriy Shopkhoev, Ammar Ali, Magauiya Zhussip +4
We introduce ReplaceMe, a generalized training-free depth pruning method that effectively replaces transformer blocks with a linear operation, while maintaining high performance fo…
DRAGOn: Designing RAG On Periodically Updated Corpus
Fedor Chernogorskii, Sergei Averkiev, Liliya Kudraleeva +4
This paper introduces DRAGOn, method to design a RAG benchmark on a regularly updated corpus. It features recent reference datasets, a question generation framework, an automatic e…
Hierarchical Embedding Fusion for Retrieval-Augmented Code Generation
Nikita Sorokin, Ivan Sedykh, Valentin Malykh
Retrieval-augmented code generation often conditions the decoder on large retrieved code snippets. This ties online inference cost to repository size and introduces noise from long…
MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks
Artem Chervyakov, Alexander Kharitonov, Pavel Zadorozhny +20
Advancements in LLMs have enhanced task automation in software engineering; however, current evaluations primarily focus on natural language tasks, overlooking code quality. Most b…