16 citations · 24 across the 4 of their papers we have counts for
3 papers · 1 filter
Of Non-Linearity and Commutativity in BERT
Sumu Zhao, Damian Pascual, Gino Brunner +1
In this work we provide new insights into the transformer architecture, and in particular, its best-known variant, BERT. First, we propose a method to measure the degree of non-lin…
On Identifiability in Transformers
Gino Brunner, Yang Liu, Damián Pascual +3
In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the ident…
Natural Language Multitasking: Analyzing and Improving Syntactic Saliency of Hidden Representations
Gino Brunner, Yuyi Wang, Roger Wattenhofer +1
We train multi-task autoencoders on linguistic tasks and analyze the learned hidden sentence representations. The representations change significantly when translation and part-of-…