15 citations · 20 across the 7 of their papers we have counts for
8 papers
WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents
Anton Emelyanov, Maria Tikhonova, Zaven Martirosian +2
We present WebPageBench, an open framework for evaluating web agents in which every task is verified from the interface's own event log. Six instrumented mock sites with brand iden…
FiMMIA: scaling semantic perturbation-based membership inference across modalities
Anton Emelyanov, Sergei Kudriashov, Alena Fenogenova
Membership Inference Attacks (MIAs) aim to determine whether a specific data point was included in the training set of a target model. Although there are have been numerous methods…
Multimodal Evaluation of Russian-language Architectures
Artem Chervyakov, Ulyana Isaeva, Anton Emelyanov +15
Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, a…
MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks
Artem Chervyakov, Alexander Kharitonov, Pavel Zadorozhny +20
Advancements in LLMs have enhanced task automation in software engineering; however, current evaluations primarily focus on natural language tasks, overlooking code quality. Most b…
MERA: A Comprehensive LLM Evaluation in Russian
Alena Fenogenova, Artem Chervyakov, Nikita Martynov +16
Over the past few years, one of the most notable advancements in AI research has been in foundation models (FMs), headlined by the rise of language models (LMs). As the models' siz…
RuCLIP -- new models and experiments: a technical report
Alex Shonenkov, Andrey Kuznetsov, Denis Dimitrov +10
In the report we propose six new implementations of ruCLIP model trained on our 240M pairs. The accuracy results are compared with original CLIP model with Ru-En translation (OPUS-…
Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models
Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov +6
In the last year, new neural architectures and multilingual pre-trained models have been released for Russian, which led to performance evaluation problems across a range of langua…
Multilingual Named Entity Recognition Using Pretrained Embeddings, Attention Mechanism and NCRF
Anton A. Emelyanov, Ekaterina Artemova
In this paper we tackle multilingual named entity recognition task. We use the BERT Language Model as embeddings with bidirectional recurrent network, attention, and NCRF on the to…