Publications (6)
Multimodal Evaluation of Russian-language Architectures
Artem Chervyakov, Ulyana Isaeva, Anton Emelyanov +15
Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, a…
Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models
Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov +6
In the last year, new neural architectures and multilingual pre-trained models have been released for Russian, which led to performance evaluation problems across a range of langua…
MERA: A Comprehensive LLM Evaluation in Russian
Alena Fenogenova, Artem Chervyakov, Nikita Martynov +16
Over the past few years, one of the most notable advancements in AI research has been in foundation models (FMs), headlined by the rise of language models (LMs). As the models' siz…
TAPE: Assessing Few-shot Russian Language Understanding
Ekaterina Taktasheva, Tatiana Shavrina, Alena Fenogenova +11
Recent advances in zero-shot and few-shot learning have shown promise for a scope of research and practical purposes. However, this fast-growing area lacks standardized evaluation…
RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov +7
In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE. Recent advances in the field of universal language models and tr…
Long Input Benchmark for Russian Analysis
Igor Churin, Murat Apishev, Maria Tikhonova +5
Recent advancements in Natural Language Processing (NLP) have fostered the development of Large Language Models (LLMs) that can solve an immense variety of tasks. One of the key as…