77 citations · 170 across the 8 of their papers we have counts for
12 papers
Of Non-Linearity and Commutativity in BERT
Sumu Zhao, Damian Pascual, Gino Brunner +1
In this work we provide new insights into the transformer architecture, and in particular, its best-known variant, BERT. First, we propose a method to measure the degree of non-lin…
Medley2K: A Dataset of Medley Transitions
Lukas Faber, Sandro Luck, Damian Pascual +3
The automatic generation of medleys, i.e., musical pieces formed by different songs concatenated via smooth transitions, is not well studied in the current literature. To facilitat…
Telling BERT's full story: from Local Attention to Global Aggregation
Damian Pascual, Gino Brunner, Roger Wattenhofer
We take a deep look into the behavior of self-attention heads in the transformer architecture. In light of recent work discouraging the use of attention distributions for explainin…
On Identifiability in Transformers
Gino Brunner, Yang Liu, Damián Pascual +3
In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the ident…
Attentive Multi-Task Deep Reinforcement Learning
Timo Bram, Gino Brunner, Oliver Richter +1
Sharing knowledge between tasks is vital for efficient learning in a multi-task setting. However, most research so far has focused on the easier case where knowledge transfer is no…
Using State Predictions for Value Regularization in Curiosity Driven Deep Reinforcement Learning
Gino Brunner, Manuel Fritsche, Oliver Richter +1
Learning in sparse reward settings remains a challenge in Reinforcement Learning, which is often addressed by using intrinsic rewards. One promising strategy is inspired by human c…