26 citations · 56 across the 7 of their papers we have counts for
7 papers
Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Alexander Borzunov, Max Ryabinin, Artem Chumachenko +5
Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using thes…
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian +6
Recent advances in large language model (LLM) pretraining have led to high-quality LLMs with impressive abilities. By compressing such LLMs via quantization to 3-4 bits per paramet…
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient
Max Ryabinin, Tim Dettmers, Michael Diskin +1
Many deep learning applications benefit from using large models with billions of parameters. Training these models is notoriously expensive due to the need for specialized HPC clus…
Petals: Collaborative Inference and Fine-tuning of Large Models
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers +5
Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download…
Training Transformers Together
Alexander Borzunov, Max Ryabinin, Tim Dettmers +5
The infrastructure necessary for training state-of-the-art models is becoming overly expensive, which makes training such models affordable only to large corporations and instituti…
Secure Distributed Training at Scale
Eduard Gorbunov, Alexander Borzunov, Michael Diskin +1
Many areas of deep learning benefit from using increasingly larger neural networks trained on public data, as is the case for pre-trained models for NLP and computer vision. Traini…
Distributed Deep Learning in Open Collaborations
Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin +13
Modern deep learning applications require increasingly more compute to train state-of-the-art models. To address this demand, large corporations and institutions use dedicated High…