Publications (34)
Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets
Max Ryabinin, Andrey Malinin, Mark Gales
Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk +1
SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
Ruslan Svirschevski, Avner May, Zhuoming Chen +3
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
Zhuoming Chen, Avner May, Ruslan Svirschevski +4
INTELLECT-1 Technical Report
Sami Jaghouar, Jack Min Ong, Manveer Basra +9
Secure Distributed Training at Scale
Eduard Gorbunov, Alexander Borzunov, Michael Diskin +1
Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements
Anton Voronov, Lena Wolf, Max Ryabinin
TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Jack Min Ong, Matthew Di Ferrante, Aaron Pazdera +5
Label Privacy in Split Learning for Large Models with Parameter-Efficient Training
Philip Zmushko, Marat Mansurov, Ruslan Svirschevski +3
Petals: Collaborative Inference and Fine-tuning of Large Models
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers +5
Is This Loss Informative? Faster Text-to-Image Customization by Tracking Objective Dynamics
Anton Voronov, Mikhail Khoroshikh, Artem Babenko +1
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop, :, Teven Le Scao +391
Asynchronous Reasoning: Training-Free Interactive Thinking LLMs
George Yakushev, Nataliia Babina, Masoud Vahid Dastgerdi +4
Mashup Learning: Faster Finetuning by Remixing Past Checkpoints
Sofia Maria Lo Cicero Vaina, Artem Chumachenko, Max Ryabinin
FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models
Ionut-Vlad Modoranu, Mher Safaryan, Erik Schultheis +3
Multilingual Language Model Pretraining using Machine-translated Data
Jiayi Wang, Yao Lu, Maurice Weber +5
Distributed Deep Learning in Open Collaborations
Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin +13
Towards Best Practices for Open Datasets for LLM Training
Stefan Baack, Stella Biderman, Kasia Odrozek +36
Embedding Words in Non-Vector Space with Unsupervised Graph Learning
Max Ryabinin, Sergei Popov, Liudmila Prokhorenkova +1
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient
Max Ryabinin, Tim Dettmers, Michael Diskin +1
Hypernymy Understanding Evaluation of Text-to-Image Models via WordNet Hierarchy
Anton Baryshnikov, Max Ryabinin
It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning
Alexey Tikhonov, Max Ryabinin
Training Transformers Together
Alexander Borzunov, Max Ryabinin, Tim Dettmers +5
AutoJudge: Judge Decoding Without Manual Annotation
Roman Garipov, Fedor Velikonivtsev, Ivan Ermakov +3
RedPajama: an Open Dataset for Training Large Language Models
Maurice Weber, Daniel Fu, Quentin Anthony +16
RuCoLA: Russian Corpus of Linguistic Acceptability
Vladislav Mikhailov, Tatiana Shamardina, Max Ryabinin +3
Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guarantees
Aleksandr Beznosikov, Peter Richtárik, Michael Diskin +2
Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Alexander Borzunov, Max Ryabinin, Artem Chumachenko +5
Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Language
Jiayi Wang, Yao Lu, Maurice Weber +4
The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models
Giwon Hong, Aryo Pradipta Gema, Rohit Saxena +8
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Ying Sheng, Lianmin Zheng, Binhang Yuan +11
Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
Ravi Ghadia, Maksim Abraham, Sergei Vorobyov +1
Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
Max Ryabinin, Anton Gusev
Escaping the Verifier: Learning to Reason via Demonstrations
Locke Cai, Max Ryabinin, Ivan Provilkov