Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders

Yitong Jiang, Hongjun Wang, Collin McCarthy +15

Vision foundation models are bottlenecked by the quadratic cost of self-attention, which limits usable resolution and increases the cost of large-scale pretraining. Subquadratic al…