10 papers
PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapters
Krishu K Thapa, Reet Barik, Krishna Teja Chitty-Venkata +2
Training large models ranging from millions to billions of parameters is highly resource-intensive, requiring significant time, compute, and memory. It is observed that most of the…
Swimba: Switch Mamba Model Scales State Space Models
Zhixu Du, Krishna Teja Chitty-Venkata, Murali Emani +3
Mixture-of-experts (MoE) is a common approach for increasing parameter capacity, but applying MoE to state space model (SSM) token mixers can multiply the cost of the recurrent sta…
ImageNet-Think-250K: A Large-Scale Synthetic Dataset for Multimodal Reasoning for Vision Language Models
Krishna Teja Chitty-Venkata, Murali Emani
We develop ImageNet-Think, a multimodal reasoning dataset designed to aid the development of Vision Language Models (VLMs) with explicit reasoning capabilities. Our dataset is buil…
PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
Krishna Teja Chitty-Venkata, Jie Ye, Xian-He Sun +4
KV caching significantly improves the efficiency of Large Language Model (LLM) inference by storing attention states from previously processed tokens, enabling faster generation of…
LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
Krishna Teja Chitty-Venkata, Sandeep Madireddy, Murali Emani +1
Mixture-of-Experts (MoE) models scale efficiently by activating only a subset of experts per token, offering a computationally sparse alternative to dense architectures. While prio…
MoPEQ: Mixture of Mixed Precision Quantized Experts
Krishna Teja Chitty-Venkata, Jie Ye, Murali Emani
Large Language and Vision Models using a Mixture-of-Experts (MoE) architecture pose significant challenges for deployment due to their computational and memory demands. Mixed Preci…