30 citations · 58 across the 8 of their papers we have counts for
10 papers
Can BERT eat RuCoLA? Topological Data Analysis to Explain
Irina Proskurina, Irina Piontkovskaya, Ekaterina Artemova
This paper investigates how Transformer language models (LMs) fine-tuned for acceptability classification capture linguistic features. Our approach uses the best practices of topol…
PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing
Xiaozhe Ren, Pingyi Zhou, Xinfan Meng +14
The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-param…
Topological Data Analysis for Speech Processing
Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva +5
We apply topological data analysis (TDA) to speech classification problems and to the introspection of a pretrained speech model, HuBERT. To this end, we introduce a number of topo…
Betti numbers of attention graphs is all you really need
Laida Kushnareva, Dmitri Piontkovski, Irina Piontkovskaya
We apply methods of topological analysis to the attention graphs, calculated on the attention heads of the BERT model ( arXiv:1810.04805v2 ). Our research shows that the classifier…
Template-based Approach to Zero-shot Intent Recognition
Dmitry Lamanov, Pavel Burnyshev, Ekaterina Artemova +3
The recent advances in transfer learning techniques and pre-training of large contextualized encoders foster innovation in real-life applications, including dialog assistants. Prac…
Acceptability Judgements via Examining the Topology of Attention Maps
Daniil Cherniavskii, Eduard Tulchinskii, Vladislav Mikhailov +7
The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP. However, the ability of the attention heads to judge the grammatical acce…