121 citations · 195 across the 7 of their papers we have counts for
12 papers · 1 filter
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers
Michael Hassid, Hao Peng, Daniel Rotem +4
The attention mechanism is considered the backbone of the widely-used Transformer architecture. It contextualizes the input by computing input-specific attention matrices. We find…
Modeling Context With Linear Attention for Scalable Document-Level Translation
Zhaofeng Wu, Hao Peng, Nikolaos Pappas +1
Document-level machine translation leverages inter-sentence dependencies to produce more coherent and consistent translations. However, these models, predominantly based on transfo…
XLTime: A Cross-Lingual Knowledge Transfer Framework for Temporal Expression Extraction
Yuwei Cao, William Groves, Tanay Kumar Saha +4
Temporal Expression Extraction (TEE) is essential for understanding time in natural language. It has applications in Natural Language Processing (NLP) tasks such as question answer…
Random Feature Attention
Hao Peng, Nikolaos Pappas, Dani Yogatama +3
Transformers are state-of-the-art models for a variety of sequence modeling tasks. At their core is an attention function which models pairwise interactions between the inputs at e…
Finetuning Pretrained Transformers into RNNs
Jungo Kasai, Hao Peng, Yizhe Zhang +6
Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. But this comes with a significant computational cost, as the attention mechanism's c…
Contextualized Perturbation for Textual Adversarial Attack
Dianqi Li, Yizhe Zhang, Hao Peng +4
Adversarial examples expose the vulnerabilities of natural language processing (NLP) models, and can be used to evaluate and improve their robustness. Existing techniques of genera…