203 citations · 593 across the 26 of their papers we have counts for
5 papers · 1 filter
TRANSOM: An Efficient Fault-Tolerant System for Training LLMs
Baodong Wu, Lei Xia, Qingping Li +6
Large language models (LLMs) with hundreds of billions or trillions of parameters, represented by chatGPT, have achieved profound impact on various fields. However, training LLMs w…
A High-Performance Design, Implementation, Deployment, and Evaluation of The Slim Fly Network
Nils Blach, Maciej Besta, Daniele De Sensi +10
Novel low-diameter network topologies such as Slim Fly (SF) offer significant cost and power advantages over the established Fat Tree, Clos, or Dragonfly. To spearhead the adoption…
Co-design Hardware and Algorithm for Vector Search
Wenqi Jiang, Shigang Li, Yu Zhu +8
Vector search has emerged as the foundation for large-scale information retrieval and machine learning systems, with search engines like Google and Bing processing tens of thousand…
ASDL: A Unified Interface for Gradient Preconditioning in PyTorch
Kazuki Osawa, Satoki Ishikawa, Rio Yokota +2
Gradient preconditioning is a key technique to integrate the second-order information into gradients for improving and extending gradient-based learning algorithms. In deep learnin…
AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost
Jinfan Chen, Shigang Li, Ran Gun +2
Recent advances in deep learning are driven by the growing scale of computation, data, and models. However, efficiently training large-scale models on distributed systems requires…