4 papers
Down and Across: Introducing Crossword-Solving as a New NLP Benchmark
Saurabh Kulshreshtha, Olga Kovaleva, Namrata Shivagunde +1
Solving crossword puzzles requires diverse reasoning capabilities, access to a vast amount of knowledge about language and the world, and the ability to satisfy the constraints imp…
BERT Busters: Outlier Dimensions that Disrupt Transformers
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers +1
Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisin…
A Primer in BERTology: What we know about how BERT works
Anna Rogers, Olga Kovaleva, Anna Rumshisky
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey o…
Revealing the Dark Secrets of BERT
Olga Kovaleva, Alexey Romanov, Anna Rogers +1
BERT-based architectures currently give state-of-the-art performance on many NLP tasks, but little is known about the exact mechanisms that contribute to its success. In the curren…