Manifold-Preserving Transformers are Effective for Short-Long Range Encoding

Ayan Sengupta, Md Shad Akhtar, Tanmoy Chakraborty

Multi-head self-attention-based Transformers have shown promise in different learning tasks. Albeit these models exhibit significant improvement in understanding short-term and lon…