4 citations · 5 across the 14 of their papers we have counts for
14 papers
KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMs
Maryam Dialameh, Hossein Rajabzadeh, Harish Krishnamoorthy Murali +3
Pipeline parallelism (PP) is widely used to scale large language model (LLM) training, but its efficiency is often limited by stage imbalance and pipeline bubbles. Meanwhile, cross…
CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training
Rezaul Karim, Austin Wen, Wang Zongzuo +3
The rapid growth in the size of large language models has necessitated the partitioning of computational workloads across accelerators such as GPUs, TPUs, and NPUs. However, these…
Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide
Hossam Amer, Rezaul Karim, Ali Pourranjbar +3
With the rapid growth of large language models (LLMs), a wide range of methods have been developed to distribute computation and memory across hardware devices for efficient traini…
FLOP-Efficient Training: Early Stopping Based on Test-Time Compute Awareness
Hossam Amer, Maryam Dialameh, Hossein Rajabzadeh +3
Scaling training compute, measured in FLOPs, has long been shown to improve the accuracy of large language models, yet training remains resource-intensive. Prior work shows that in…
Bayesian Mixture of Experts For Large Language Models
Maryam Dialameh, Hossein Rajabzadeh, Weiwei Zhang +2
We present Bayesian Mixture of Experts (Bayesian-MoE), a post-hoc uncertainty estimation framework for fine-tuned large language models (LLMs) based on Mixture-of-Experts architect…
Do LLMs Align with My Task? Evaluating Text-to-SQL via Dataset Alignment
Davood Rafiei, Morgan Lindsay Heisler, Weiwei Zhang +2
Supervised Fine-Tuning (SFT) is an effective method for adapting Large Language Models (LLMs) on downstream tasks. However, variability in training data can hinder a model's abilit…
Rethinking Schema Linking: A Context-Aware Bidirectional Retrieval Approach for Text-to-SQL
Md Mahadi Hasan Nahid, Davood Rafiei, Weiwei Zhang +1
Schema linking -- the process of aligning natural language questions with database schema elements -- is a critical yet underexplored component of Text-to-SQL systems. While recent…
TaoCache: Structure-Maintained Video Generation Acceleration
Zhentao Fan, Zongzuo Wang, Weiwei Zhang
Existing cache-based acceleration methods for video diffusion models primarily skip early or mid denoising steps, which often leads to structural discrepancies relative to full-tim…
Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection
Mohammad Mahdi Moradi, Hossam Amer, Sudhir Mudur +3
Learning to adapt pretrained language models to unlabeled, out-of-distribution data is a critical challenge, as models often falter on structurally novel reasoning tasks even while…
Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks
Mohammad Mahdi Moradi, Walid Ahmed, Shuangyue Wen +3
Attention and State-Space Models (SSMs) when combined in a hybrid network in sequence or in parallel provide complementary strengths. In a hybrid sequential pipeline they alternate…
Enhancing Learned Knowledge in LoRA Adapters Through Efficient Contrastive Decoding on Ascend NPUs
Morgan Lindsay Heisler, Linzi Xing, Ge Shi +7
Huawei Cloud users leverage LoRA (Low-Rank Adaptation) as an efficient and scalable method to fine-tune and customize large language models (LLMs) for application-specific needs. H…
DeTriever: Decoder-representation-based Retriever for Improving NL2SQL In-Context Learning
Yuxi Feng, Raymond Li, Zhenan Fan +4
While in-context Learning (ICL) has proven to be an effective technique to improve the performance of Large Language Models (LLMs) in a variety of complex tasks, notably in transla…
SQL-Encoder: Improving NL2SQL In-Context Learning Through a Context-Aware Encoder
Mohammadreza Pourreza, Davood Rafiei, Yuxi Feng +3
Detecting structural similarity between queries is essential for selecting examples in in-context learning models. However, assessing structural similarity based solely on the natu…
Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling
Jixuan Wang, Kai Wei, Martin Radfar +2
We propose a novel Transformer encoder-based architecture with syntactical knowledge encoded for intent detection and slot filling. Specifically, we encode syntactic knowledge into…