Input Compression with Positional Consistency for Efficient Training and Inference of Transformer Neural Networks

Amrit Nagarajan, Anand Raghunathan

Transformers have rapidly increased in popularity in recent years, achieving state-of-the-art performance in processing text, images, audio and video. However, Transformers present…