77 citations · 106 across the 9 of their papers we have counts for
13 papers
pNLP-Mixer: an Efficient all-MLP Architecture for Language
Francesco Fusco, Damian Pascual, Peter Staar +1
Large pre-trained language models based on transformer architecture have drastically changed the natural language processing (NLP) landscape. However, deploying those models for on…
EEGEyeNet: a Simultaneous Electroencephalography and Eye-tracking Dataset and Benchmark for Eye Movement Prediction
Ard Kastrati, Martyna Beata Płomecka, Damián Pascual +4
We present a new dataset and benchmark with the goal of advancing research in the intersection of brain activities and eye movements. Our dataset, EEGEyeNet, consists of simultaneo…
On Isotropy Calibration of Transformers
Yue Ding, Karolis Martinkus, Damian Pascual +2
Different studies of the embedding space of transformer models suggest that the distribution of contextual representations is highly anisotropic - the embeddings are distributed in…
A Plug-and-Play Method for Controlled Text Generation
Damian Pascual, Beni Egressy, Clara Meister +2
Large pre-trained language models have repeatedly shown their ability to produce fluent text. Yet even when starting from a prompt, generation can continue in many plausible direct…
BERT is Robust! A Case Against Synonym-Based Adversarial Examples in Text Classification
Jens Hauser, Zhao Meng, Damián Pascual +1
Deep Neural Networks have taken Natural Language Processing by storm. While this led to incredible improvements across many tasks, it also initiated a new research field, questioni…
Towards BERT-based Automatic ICD Coding: Limitations and Opportunities
Damian Pascual, Sandro Luck, Roger Wattenhofer
Automatic ICD coding is the task of assigning codes from the International Classification of Diseases (ICD) to medical notes. These codes describe the state of the patient and have…
Of Non-Linearity and Commutativity in BERT
Sumu Zhao, Damian Pascual, Gino Brunner +1
In this work we provide new insights into the transformer architecture, and in particular, its best-known variant, BERT. First, we propose a method to measure the degree of non-lin…
Directed Beam Search: Plug-and-Play Lexically Constrained Language Generation
Damian Pascual, Beni Egressy, Florian Bolli +1
Large pre-trained language models are capable of generating realistic text. However, controlling these models so that the generated text satisfies lexical constraints, i.e., contai…
Brain2Word: Decoding Brain Activity for Language Generation
Nicolas Affolter, Beni Egressy, Damian Pascual +1
Brain decoding, understood as the process of mapping brain activities to the stimuli that generated them, has been an active research area in the last years. In the case of languag…
Medley2K: A Dataset of Medley Transitions
Lukas Faber, Sandro Luck, Damian Pascual +3
The automatic generation of medleys, i.e., musical pieces formed by different songs concatenated via smooth transitions, is not well studied in the current literature. To facilitat…
Telling BERT's full story: from Local Attention to Global Aggregation
Damian Pascual, Gino Brunner, Roger Wattenhofer
We take a deep look into the behavior of self-attention heads in the transformer architecture. In light of recent work discouraging the use of attention distributions for explainin…
On Identifiability in Transformers
Gino Brunner, Yang Liu, Damián Pascual +3
In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the ident…
Synthetic Epileptic Brain Activities Using Generative Adversarial Networks
Damian Pascual, Amir Aminifar, David Atienza +2
Epilepsy is a chronic neurological disorder affecting more than 65 million people worldwide and manifested by recurrent unprovoked seizures. The unpredictability of seizures not on…