49 citations · 201 across the 21 of their papers we have counts for
24 papers
REPA: Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilities
Alexander Pugachev, Alena Fenogenova, Vladislav Mikhailov +1
Recent advances in large language models (LLMs) have introduced the novel paradigm of using LLMs as judges, where an LLM evaluates and scores the outputs of another LLM, which ofte…
GenAI Content Detection Task 1: English and Multilingual Machine-Generated Text Detection: AI vs. Human
Yuxia Wang, Artem Shelmanov, Jonibek Mansurov +23
We present the GenAI Content Detection Task~1 -- a shared task on binary machine generated text detection, conducted as a part of the GenAI workshop at COLING 2025. The task consis…
Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI
Yuxia Wang, Rui Xing, Jonibek Mansurov +23
Prior studies have shown that distinguishing text generated by Large Language Models (LLMs) from human-written one is highly challenging for humans, and often no better than random…
Beemo: Benchmark of Expert-edited Machine-generated Outputs
Ekaterina Artemova, Jason Lucas, Saranya Venkatraman +4
The rapid proliferation of large language models (LLMs) has increased the volume of machine-generated texts (MGTs) and blurred text authorship in various domains. However, most exi…
LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection
Mervat Abassy, Kareem Elozeiri, Alexander Aziz +21
The ease of access to large language models (LLMs) has enabled a widespread of machine-generated texts, and now it is often hard to tell whether a piece of text was human-written o…
Papilusion at DAGPap24: Paper or Illusion? Detecting AI-generated Scientific Papers
Nikita Andreev, Alexander Shirnin, Vladislav Mikhailov +1
This paper presents Papilusion, an AI-generated scientific text detector developed within the DAGPap24 shared task on detecting automatically generated scientific papers. We propos…
AIpom at SemEval-2024 Task 8: Detecting AI-produced Outputs in M4
Alexander Shirnin, Nikita Andreev, Vladislav Mikhailov +1
This paper describes AIpom, a system designed to detect a boundary between human-written and machine-generated text (SemEval-2024 Task 8, Subtask C: Human-Machine Mixed Text Detect…
LUNA: A Framework for Language Understanding and Naturalness Assessment
Marat Saidov, Aleksandra Bakalova, Ekaterina Taktasheva +2
The evaluation of Natural Language Generation (NLG) models has gained increased attention, urging the development of metrics that evaluate various aspects of generated text. LUNA a…
A Family of Pretrained Transformer Language Models for Russian
Dmitry Zmitrovich, Alexander Abramov, Andrey Kalmykov +10
Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian…
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop, :, Teven Le Scao +391
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to wi…
RuCoLA: Russian Corpus of Linguistic Acceptability
Vladislav Mikhailov, Tatiana Shamardina, Max Ryabinin +3
Linguistic acceptability (LA) attracts the attention of the research community due to its many uses, such as testing the grammatical knowledge of language models and filtering impl…
TAPE: Assessing Few-shot Russian Language Understanding
Ekaterina Taktasheva, Tatiana Shavrina, Alena Fenogenova +11
Recent advances in zero-shot and few-shot learning have shown promise for a scope of research and practical purposes. However, this fast-growing area lacks standardized evaluation…
Vote'n'Rank: Revision of Benchmarking with Social Choice Theory
Mark Rofin, Vladislav Mikhailov, Mikhail Florinskiy +5
The development of state-of-the-art systems in different applied areas of machine learning (ML) is driven by benchmarks, which have shaped the paradigm of evaluating generalisation…
Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
Tatiana Shamardina, Vladislav Mikhailov, Daniil Chernianskii +7
We present the shared task on artificial text detection in Russian, which is organized as a part of the Dialogue Evaluation initiative, held in 2022. The shared task dataset includ…
Acceptability Judgements via Examining the Topology of Attention Maps
Daniil Cherniavskii, Eduard Tulchinskii, Vladislav Mikhailov +7
The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP. However, the ability of the attention heads to judge the grammatical acce…
mGPT: Few-Shot Learners Go Multilingual
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova +3
Recent studies report that autoregressive language models can successfully solve many NLP tasks via zero- and few-shot learning paradigms, which opens up new possibilities for usin…
Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models
Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov +6
In the last year, new neural architectures and multilingual pre-trained models have been released for Russian, which led to performance evaluation problems across a range of langua…
Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations
Ekaterina Taktasheva, Vladislav Mikhailov, Ekaterina Artemova
Recent research has adopted a new experimental field centered around the concept of text perturbations which has revealed that shuffled word order has little to no impact on the do…
Artificial Text Detection via Examining the Topology of Attention Maps
Laida Kushnareva, Daniil Cherniavskii, Vladislav Mikhailov +6
The impressive capabilities of recent generative models to create texts that are challenging to distinguish from the human-written ones can be misused for generating fake news, pro…
Morph Call: Probing Morphosyntactic Content of Multilingual Transformers
Vladislav Mikhailov, Oleg Serikov, Ekaterina Artemova
The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploring their inner workings. Recent research…
MOROCCO: Model Resource Comparison Framework
Valentin Malykh, Alexander Kukushkin, Ekaterina Artemova +3
The new generation of pre-trained NLP models push the SOTA to the new limits, but at the cost of computational resources, to the point that their use in real production environment…
RuSentEval: Linguistic Source, Encoder Force!
Vladislav Mikhailov, Ekaterina Taktasheva, Elina Sigdel +1
The success of pre-trained transformer language models has brought a great deal of interest on how these models work, and what they learn about language. However, prior research in…
Domain-Transferable Method for Named Entity Recognition Task
Vladislav Mikhailov, Tatiana Shavrina
Named Entity Recognition (NER) is a fundamental task in the fields of natural language processing and information extraction. NER has been widely used as a standalone tool or an es…
RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov +7
In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE. Recent advances in the field of universal language models and tr…