Publications (32)
DRAGOn: Designing RAG On Periodically Updated Corpus
Fedor Chernogorskii, Sergei Averkiev, Liliya Kudraleeva +4
This paper introduces DRAGOn, method to design a RAG benchmark on a regularly updated corpus. It features recent reference datasets, a question generation framework, an automatic e…
Searching by Code: a New SearchBySnippet Dataset and SnippeR Retrieval Model for Searching by Code Snippets
Ivan Sedykh, Dmitry Abulkhanov, Nikita Sorokin +2
Code search is an important and well-studied task, but it usually means searching for code by a text query. We argue that using a code snippet (and possibly an error traceback) as…
AspeRa: Aspect-based Rating Prediction Model
Sergey I. Nikolenko, Elena Tutubalina, Valentin Malykh +2
We propose a novel end-to-end Aspect-based Rating Prediction model (AspeRa) that estimates user rating based on review texts for the items and at the same time discovers coherent a…
Low-resource Machine Translation for Code-switched Kazakh-Russian Language Pair
Maksim Borisov, Zhanibek Kozhirbayev, Valentin Malykh
Machine translation for low resource language pairs is a challenging task. This task could become extremely difficult once a speaker uses code switching. We propose a method to bui…
Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
Ivan Sedykh, Nikita Sorokin, Valentin Malykh
Recent advances in masked diffusion language models (MDLMs) narrow the quality gap to autoregressive LMs, but their sampling remains expensive because generation requires many full…
Answer Candidate Type Selection: Text-to-Text Language Model for Closed Book Question Answering Meets Knowledge Graphs
Mikhail Salnikov, Maria Lysyuk, Pavel Braslavski +3
Pre-trained Text-to-Text Language Models (LMs), such as T5 or BART yield promising results in the Knowledge Graph Question Answering (KGQA) task. However, the capacity of the model…
MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks
Artem Chervyakov, Alexander Kharitonov, Pavel Zadorozhny +20
Advancements in LLMs have enhanced task automation in software engineering; however, current evaluations primarily focus on natural language tasks, overlooking code quality. Most b…
AINL-Eval 2025 Shared Task: Detection of AI-Generated Scientific Abstracts in Russian
Tatiana Batura, Elena Bruches, Milana Shvenk +1
The rapid advancement of large language models (LLMs) has revolutionized text generation, making it increasingly difficult to distinguish between human- and AI-generated content. T…
SumHiS: Extractive Summarization Exploiting Hidden Structure
Tikhonov Pavel, Anastasiya Ianina, Valentin Malykh
Extractive summarization is a task of highlighting the most important parts of the text. We introduce a new approach to extractive summarization task using hidden clustering struct…
Improving unsupervised neural aspect extraction for online discussions using out-of-domain classification
Anton Alekseev, Elena Tutubalina, Valentin Malykh +1
Deep learning architectures based on self-attention have recently achieved and surpassed state of the art results in the task of unsupervised aspect extraction and topic modeling.…
WikiMulti: a Corpus for Cross-Lingual Summarization
Pavel Tikhonov, Valentin Malykh
Cross-lingual summarization (CLS) is the task to produce a summary in one particular language for a source document in a different language. We introduce WikiMulti - a new dataset…
Iterative Self-Training for Code Generation via Reinforced Re-Ranking
Nikita Sorokin, Ivan Sedykh, Valentin Malykh
Generating high-quality code that solves complex programming tasks is challenging, especially with current decoder-based models that produce highly stochastic outputs. In code gene…
The Second Conversational Intelligence Challenge (ConvAI2)
Emily Dinan, Varvara Logacheva, Valentin Malykh +14
We describe the setting and results of the ConvAI2 NeurIPS competition that aims to further the state-of-the-art in open-domain chatbots. Some key takeaways from the competition ar…
SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
Pavel Adamenko, Mikhail Ivanov, Aidar Valeev +6
The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench datas…
ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
Dmitriy Shopkhoev, Ammar Ali, Magauiya Zhussip +4
We introduce ReplaceMe, a generalized training-free depth pruning method that effectively replaces transformer blocks with a linear operation, while maintaining high performance fo…
CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement
Maria Dziuba, Valentin Malykh
Effective generation of structured code comments requires robust quality metrics for dataset curation, yet existing approaches (SIDE, MIDQ, STASIS) suffer from limited code-comment…
Prompt Embedding Probes (PEP): Hallucination Detection in LLMs from Hidden States
Zakhar Mrykhin, Valentin Malykh
Large language models (LLMs) can generate fluent and useful responses but remain prone to hallucinations. We introduce Prompt Embedding Probes (PEP), a white-box method for answer-…
CCT-Code: Cross-Consistency Training for Multilingual Clone Detection and Code Search
Anton Tikhonov, Nikita Sorokin, Dmitry Abulkhanov +3
We consider the well-known and important tasks of clone detection and information retrieval for source code. The most standard setup is to search clones inside the same language co…
A Single Example Can Improve Zero-Shot Data Generation
Pavel Burnyshev, Valentin Malykh, Andrey Bout +2
Sub-tasks of intent classification, such as robustness to distribution shift, adaptation to specific user groups and personalization, out-of-domain detection, require extensive and…
Sequence Learning with RNNs for Medical Concept Normalization in User-Generated Texts
Elena Tutubalina, Zulfat Miftahutdinov, Sergey Nikolenko +1
In this work, we consider the medical concept normalization problem, i.e., the problem of mapping a disease mention in free-form text to a concept in a controlled vocabulary, usual…
RecVAE: a New Variational Autoencoder for Top-N Recommendations with Implicit Feedback
Ilya Shenbin, Anton Alekseev, Elena Tutubalina +2
Recent research has shown the advantages of using autoencoders based on deep neural networks for collaborative filtering. In particular, the recently proposed Mult-VAE model, which…
Large Language Models Meet Knowledge Graphs to Answer Factoid Questions
Mikhail Salnikov, Hai Le, Prateek Rajput +4
Recently, it has been shown that the incorporation of structured knowledge into Large Language Models significantly improves the results for a variety of NLP tasks. In this paper,…
Template-based Approach to Zero-shot Intent Recognition
Dmitry Lamanov, Pavel Burnyshev, Ekaterina Artemova +3
The recent advances in transfer learning techniques and pre-training of large contextualized encoders foster innovation in real-life applications, including dialog assistants. Prac…
DetIE: Multilingual Open Information Extraction Inspired by Object Detection
Michael Vasilkovsky, Anton Alekseev, Valentin Malykh +6
State of the art neural methods for open information extraction (OpenIE) usually extract triplets (or tuples) iteratively in an autoregressive or predicate-based manner in order no…
IDAP++: Advancing Divergence-Based Pruning via Filter-Level and Layer-Level Optimization
Aleksei Samarin, Artem Nazarenko, Egor Kotenko +3
This paper presents a novel approach to neural network compression that addresses redundancy at both the filter and architectural levels through a unified framework grounded in inf…
StRuCom: A Novel Dataset of Structured Code Comments in Russian
Maria Dziuba, Valentin Malykh
Structured code comments in docstring format are essential for code comprehension and maintenance, but existing machine learning models for their generation perform poorly for Russ…
Self-Attentive Model for Headline Generation
Daniil Gavrilov, Pavel Kalaidin, Valentin Malykh
Headline generation is a special type of text summarization task. While the amount of available training data for this task is almost unlimited, it still remains challenging, as le…
MOROCCO: Model Resource Comparison Framework
Valentin Malykh, Alexander Kukushkin, Ekaterina Artemova +3
The new generation of pre-trained NLP models push the SOTA to the new limits, but at the cost of computational resources, to the point that their use in real production environment…
Hierarchical Embedding Fusion for Retrieval-Augmented Code Generation
Nikita Sorokin, Ivan Sedykh, Valentin Malykh
Retrieval-augmented code generation often conditions the decoder on large retrieved code snippets. This ties online inference cost to repository size and introduces noise from long…
The Russian Drug Reaction Corpus and Neural Models for Drug Reactions and Effectiveness Detection in User Reviews
Elena Tutubalina, Ilseyar Alimova, Zulfat Miftahutdinov +3
The Russian Drug Reaction Corpus (RuDReC) is a new partially annotated corpus of consumer reviews in Russian about pharmaceutical products for the detection of health-related named…
Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models
Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov +6
In the last year, new neural architectures and multilingual pre-trained models have been released for Russian, which led to performance evaluation problems across a range of langua…
RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov +7
In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE. Recent advances in the field of universal language models and tr…