10 papers
CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training
Rezaul Karim, Austin Wen, Wang Zongzuo +3
The rapid growth in the size of large language models has necessitated the partitioning of computational workloads across accelerators such as GPUs, TPUs, and NPUs. However, these…
Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide
Hossam Amer, Rezaul Karim, Ali Pourranjbar +3
With the rapid growth of large language models (LLMs), a wide range of methods have been developed to distribute computation and memory across hardware devices for efficient traini…
Rethinking Schema Linking: A Context-Aware Bidirectional Retrieval Approach for Text-to-SQL
Md Mahadi Hasan Nahid, Davood Rafiei, Weiwei Zhang +1
Schema linking -- the process of aligning natural language questions with database schema elements -- is a critical yet underexplored component of Text-to-SQL systems. While recent…
FLOP-Efficient Training: Early Stopping Based on Test-Time Compute Awareness
Hossam Amer, Maryam Dialameh, Hossein Rajabzadeh +3
Scaling training compute, measured in FLOPs, has long been shown to improve the accuracy of large language models, yet training remains resource-intensive. Prior work shows that in…
Bayesian Mixture of Experts For Large Language Models
Maryam Dialameh, Hossein Rajabzadeh, Weiwei Zhang +2
We present Bayesian Mixture of Experts (Bayesian-MoE), a post-hoc uncertainty estimation framework for fine-tuned large language models (LLMs) based on Mixture-of-Experts architect…
Do LLMs Align with My Task? Evaluating Text-to-SQL via Dataset Alignment
Davood Rafiei, Morgan Lindsay Heisler, Weiwei Zhang +2
Supervised Fine-Tuning (SFT) is an effective method for adapting Large Language Models (LLMs) on downstream tasks. However, variability in training data can hinder a model's abilit…