PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapters

Krishu K Thapa, Reet Barik, Krishna Teja Chitty-Venkata +2

Training large models ranging from millions to billions of parameters is highly resource-intensive, requiring significant time, compute, and memory. It is observed that most of the…