Full-bandwidth transformer

Xi Wang, Ziyang Cai, Zheng Zhan +5

Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal acce…