4 papers · 1 filter
NEST: Network- and Memory-Aware Device Placement For Distributed Deep Learning
Irene Wang, Vishnu Varma Venkata, Arvind Krishnamurthy +1
The growing scale of deep learning demands distributed training frameworks that jointly reason about parallelism, memory, and network topology. Prior works often rely on heuristic…
Flashlight: PyTorch Compiler Extensions to Accelerate Attention Variants
Bozhi You, Irene Wang, Zelal Su Mustafaoglu +5
Attention is a fundamental building block of large language models (LLMs), so there have been many efforts to implement it efficiently. For example, FlashAttention leverages tiling…
CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization
Irene Wang, Newsha Ardalani, Mostafa Elhoushi +6
Machine learning solutions are rapidly adopted to enable a variety of key use cases, from conversational AI assistants to scientific discovery. This growing adoption is expected to…
Integrated Hardware Architecture and Device Placement Search
Irene Wang, Jakub Tarnawski, Amar Phanishayee +1
Distributed execution of deep learning training involves a dynamic interplay between hardware accelerator architecture and device placement strategy. This is the first work to expl…