30 citations · 59 across the 9 of their papers we have counts for
9 papers · 1 filter
Can BERT eat RuCoLA? Topological Data Analysis to Explain
Irina Proskurina, Irina Piontkovskaya, Ekaterina Artemova
This paper investigates how Transformer language models (LMs) fine-tuned for acceptability classification capture linguistic features. Our approach uses the best practices of topol…
PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing
Xiaozhe Ren, Pingyi Zhou, Xinfan Meng +14
The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-param…
Betti numbers of attention graphs is all you really need
Laida Kushnareva, Dmitri Piontkovski, Irina Piontkovskaya
We apply methods of topological analysis to the attention graphs, calculated on the attention heads of the BERT model ( arXiv:1810.04805v2 ). Our research shows that the classifier…
Template-based Approach to Zero-shot Intent Recognition
Dmitry Lamanov, Pavel Burnyshev, Ekaterina Artemova +3
The recent advances in transfer learning techniques and pre-training of large contextualized encoders foster innovation in real-life applications, including dialog assistants. Prac…
Acceptability Judgements via Examining the Topology of Attention Maps
Daniil Cherniavskii, Eduard Tulchinskii, Vladislav Mikhailov +7
The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP. However, the ability of the attention heads to judge the grammatical acce…
Artificial Text Detection via Examining the Topology of Attention Maps
Laida Kushnareva, Daniil Cherniavskii, Vladislav Mikhailov +6
The impressive capabilities of recent generative models to create texts that are challenging to distinguish from the human-written ones can be misused for generating fake news, pro…
A Single Example Can Improve Zero-Shot Data Generation
Pavel Burnyshev, Valentin Malykh, Andrey Bout +2
Sub-tasks of intent classification, such as robustness to distribution shift, adaptation to specific user groups and personalization, out-of-domain detection, require extensive and…
Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection
Alexander Podolskiy, Dmitry Lipin, Andrey Bout +2
Real-life applications, heavily relying on machine learning, such as dialog systems, demand out-of-domain detection methods. Intent classification models should be equipped with a…
Differentially Private Distributed Learning for Language Modeling Tasks
Vadim Popov, Mikhail Kudinov, Irina Piontkovskaya +2
One of the big challenges in machine learning applications is that training data can be different from the real-world data faced by the algorithm. In language modeling, users' lang…