215 citations · 472 across the 24 of their papers we have counts for
7 papers · 1 filter
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers
Michael Hassid, Hao Peng, Daniel Rotem +4
The attention mechanism is considered the backbone of the widely-used Transformer architecture. It contextualizes the input by computing input-specific attention matrices. We find…
Efficient Methods for Natural Language Processing: A Survey
Marcos Treviso, Ji-Ung Lee, Tianchu Ji +19
Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance mea…
WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models
Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef +4
While vision-and-language models perform well on tasks such as visual question answering, they struggle when it comes to basic human commonsense reasoning skills. In this work, we…
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias
Yarden Tal, Inbal Magar, Roy Schwartz
The size of pretrained models is increasing, and so is their performance on a variety of NLP tasks. However, as their memorization capacity grows, they might pick up more social bi…
Measuring the Carbon Intensity of AI in Cloud Instances
Jesse Dodge, Taylor Prewitt, Remi Tachet Des Combes +7
By providing unprecedented access to computational resources, cloud computing has enabled rapid growth in technologies such as machine learning, the computational demands of which…
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations
Roy Schwartz, Gabriel Stanovsky
Recent work has shown that deep learning models in NLP are highly sensitive to low-level correlations between simple features and specific output labels, leading to overfitting and…
Data Contamination: From Memorization to Exploitation
Inbal Magar, Roy Schwartz
Pretrained language models are typically trained on massive web-based datasets, which are often "contaminated" with downstream test sets. It is not clear to what extent models expl…