36 citations · 56 across the 7 of their papers we have counts for
8 papers
MegaFlux: Skew-Resilient MoE Megakernels via Pipelined Expert Replication
Jianzhu Yao, Siva Kumar Sastry Hari, Vignesh Balaji +4
Mixture-of-experts (MoE) megakernels fuse expert-parallel communication with expert computation. However, under fixed expert placement, routing skew creates GPU stragglers: overloa…
Addressing Variable Heterogeneity in Distributed Multimodal Training with Entrain
Insu Jang, Mosharaf Chowdhury
Multimodal LLM datasets are inherently heterogeneous, with significant data variability. Although each modality exhibits independent variability, sample-level entanglement makes it…
MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search
Minkyoung Cho, Insu Jang, Shuowei Jin +5
Fine-tuning Multimodal Large Language Models (MLLMs) with parameter-efficient methods like Low-Rank Adaptation (LoRA) is crucial for task adaptation. However, imbalanced training d…
Evaluation Framework for AI Systems in "the Wild"
Sarah Jabbour, Trenton Chang, Anindya Das Antar +13
Generative AI (GenAI) models have become vital across industries, yet current evaluation methods have not adapted to their widespread use. Traditional evaluations often rely on ben…
Efficient Distributed MLLM Training with Cornstarch
Insu Jang, Runyu Lu, Nikhil Bansal +2
Multimodal large language models (MLLMs) extend the capabilities of large language models (LLMs) by combining heterogeneous model architectures to handle diverse modalities like im…
Mordal: Automated Pretrained Model Selection for Vision Language Models
Shiqi He, Insu Jang, Mosharaf Chowdhury
Incorporating multiple modalities into large language models (LLMs) is a powerful way to enhance their understanding of non-textual data, enabling them to perform multimodal tasks.…
Reducing Energy Bloat in Large Model Training
Jae-Won Chung, Yile Gu, Insu Jang +3
Training large AI models on numerous GPUs consumes a massive amount of energy, making power delivery one of the largest limiting factors in building and operating datacenters for A…
Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates
Insu Jang, Zhenning Yang, Zhen Zhang +2
Oobleck enables resilient distributed training of large DNN models with guaranteed fault tolerance. It takes a planning-execution co-design approach, where it first generates a set…