49 citations · 83 across the 7 of their papers we have counts for
8 papers · 1 filter
CausalLM is not optimal for in-context learning
Nan Ding, Tomer Levinboim, Jialin Wu +2
Recent empirical evidence indicates that transformer based in-context learning performs better when using a prefix language model (prefixLM), in which in-context samples can all at…
Improving Robust Generalization by Direct PAC-Bayesian Bound Minimization
Zifan Wang, Nan Ding, Tomer Levinboim +2
Recent research in robust optimization has shown an overfitting-like phenomenon in which models trained against adversarial attacks exhibit higher robustness on the training set co…
Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning
Nan Ding, Xi Chen, Tomer Levinboim +2
Despite recent advances in its theoretical understanding, there still remains a significant gap in the ability of existing PAC-Bayesian theories on meta-learning to explain perform…
Do Transformer Modifications Transfer Across Implementations and Applications?
Sharan Narang, Hyung Won Chung, Yi Tay +13
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread…
Attention that does not Explain Away
Nan Ding, Xinjie Fan, Zhenzhong Lan +2
Models based on the Transformer architecture have achieved better accuracy than the ones based on competing architectures for a large set of tasks. A unique feature of the Transfor…
Talking-Heads Attention
Noam Shazeer, Zhenzhong Lan, Youlong Cheng +2
We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the…