2 citations · 4 across the 17 of their papers we have counts for
9 papers · 1 filter
CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training
Rezaul Karim, Austin Wen, Wang Zongzuo +3
The rapid growth in the size of large language models has necessitated the partitioning of computational workloads across accelerators such as GPUs, TPUs, and NPUs. However, these…
Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide
Hossam Amer, Rezaul Karim, Ali Pourranjbar +3
With the rapid growth of large language models (LLMs), a wide range of methods have been developed to distribute computation and memory across hardware devices for efficient traini…
EPAS: Efficient Training with Progressive Activation Sharing
Rezaul Karim, Maryam Dialameh, Yang Liu +2
We present a novel method for Efficient training with Progressive Activation Sharing (EPAS). This method bridges progressive training paradigm with the phenomenon of redundant QK (…
ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training
Maryam Dialameh, Rezaul Karim, Hossein Rajabzadeh +5
This paper introduces ECHO-LLaMA, an efficient LLaMA architecture designed to improve both the training speed and inference throughput of LLaMA architectures while maintaining its…
Accelerating the Low-Rank Decomposed Models
Habib Hajimolahoseini, Walid Ahmed, Austin Wen +1
Tensor decomposition is a mathematically supported technique for data compression. It consists of applying some kind of a Low Rank Decomposition technique on the tensors or matrice…
GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values
Farnoosh Javadi, Walid Ahmed, Habib Hajimolahoseini +7
Massive transformer-based models face several challenges, including slow and computationally intensive pre-training and over-parametrization. This paper addresses these challenges…
SwiftLearn: A Data-Efficient Training Method of Deep Learning Models using Importance Sampling
Habib Hajimolahoseini, Omar Mohamed Awad, Walid Ahmed +8
In this paper, we present SwiftLearn, a data-efficient approach to accelerate training of deep learning models using a subset of data samples selected during the warm-up stages of…
Improving Resnet-9 Generalization Trained on Small Datasets
Omar Mohamed Awad, Habib Hajimolahoseini, Michael Lim +4
This paper presents our proposed approach that won the first prize at the ICLR competition on Hardware Aware Efficient Training. The challenge is to achieve the highest possible ac…
Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantization
Habib Hajimolahoseini, Walid Ahmed, Yang Liu
Low Rank Decomposition (LRD) is a model compression technique applied to the weight tensors of deep learning models in order to reduce the number of trainable parameters and comput…