13 citations · 14 across the 2 of their papers we have counts for
6 papers · 1 filter
Introducing Orthogonal Constraint in Structural Probes
Tomasz Limisiewicz, David Mareček
With the recent success of pre-trained models in NLP, a significant focus was put on interpreting their representations. One of the most prominent approaches is structural probing…
Syntax Representation in Word Embeddings and Neural Networks -- A Survey
Tomasz Limisiewicz, David Mareček
Neural networks trained on natural language processing tasks capture syntax even though it is not provided as a supervision signal. This indicates that syntactic analysis is essent…
Inducing Syntactic Trees from BERT Representations
Rudolf Rosa, David Mareček
We use the English model of BERT and explore how a deletion of one word in a sentence changes representations of other words. Our hypothesis is that removing a reducible word (e.g.…
Derivational Morphological Relations in Word Embeddings
Tomáš Musil, Jonáš Vidra, David Mareček
Derivation is a type of a word-formation process which creates new words from existing ones by adding, changing or deleting affixes. In this paper, we explore the potential of word…
From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions
David Mareček, Rudolf Rosa
We inspect the multi-head self-attention in Transformer NMT encoders for three source languages, looking for patterns that could have a syntactic interpretation. In many of the att…
Input Combination Strategies for Multi-Source Transformer Decoder
Jindřich Libovický, Jindřich Helcl, David Mareček
In multi-source sequence-to-sequence tasks, the attention mechanism can be modeled in several ways. This topic has been thoroughly studied on recurrent architectures. In this paper…