178 citations · 321 across the 10 of their papers we have counts for
11 papers
AutoSci: A Memory-Centric Agentic System for the Full Scientific Research Lifecycle
Weitong Qian, Beicheng Xu, Zhongao Xie +16
Scientific research has traditionally been human-intensive, requiring researchers to coordinate literature, ideas, experiments, manuscripts, and review responses across long projec…
Echo: Decoupling Inference and Training for Large-Scale RL Alignment on Heterogeneous Swarms
Jie Xiao, Changyuan Fan, Qingnan Ren +6
Modern RL-based post-training for large language models (LLMs) co-locate trajectory sampling and policy optimisation on the same GPU cluster, forcing the system to switch between i…
SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
Zihao Wang, Bin Cui, Shaoduo Gan
Optimizing the Key-Value (KV) cache of the Large Language Model (LLM) has been considered critical to saving the cost of inference. Most of the existing KV-cache compression algori…
Few-shot Named Entity Recognition with Entity-level Prototypical Network Enhanced by Dispersedly Distributed Prototypes
Bin Ji, Shasha Li, Shaoduo Gan +3
Few-shot named entity recognition (NER) enables us to build a NER system for a new domain using very few labeled examples. However, existing prototypical networks for this task suf…
Stochastic Gradient Descent without Full Data Shuffle
Lijie Xu, Shuang Qiu, Binhang Yuan +9
Stochastic gradient descent (SGD) is the cornerstone of modern machine learning (ML) systems. Despite its computational efficiency, SGD requires random data access that is inherent…
FRuDA: Framework for Distributed Adversarial Domain Adaptation
Shaoduo Gan, Akhil Mathur, Anton Isopoussu +3
Breakthroughs in unsupervised domain adaptation (uDA) can help in adapting models from a label-rich source domain to unlabeled target domains. Despite these advancements, there is…
BAGUA: Scaling up Distributed Learning with System Relaxations
Shaoduo Gan, Xiangru Lian, Rui Wang +11
Recent years have witnessed a growing list of systems for distributed data-parallel training. Existing systems largely fit into two paradigms, i.e., parameter server and MPI-style…
Towards Demystifying Serverless Machine Learning Training
Jiawei Jiang, Shaoduo Gan, Yue Liu +6
The appeal of serverless (FaaS) has triggered a growing interest on how to use it in data-intensive applications such as ETL, query processing, or machine learning (ML). Several sy…
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan +6
Scalable training of large models (like BERT and GPT-3) requires careful optimization rooted in model design, architecture, and system capabilities. From a system standpoint, commu…
APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm
Hanlin Tang, Shaoduo Gan, Samyam Rajbhandari +4
Adam is the important optimization algorithm to guarantee efficiency and accuracy for training many important tasks such as BERT and ImageNet. However, Adam is generally not compat…
Communication Compression for Decentralized Training
Hanlin Tang, Shaoduo Gan, Ce Zhang +2
Optimizing distributed learning systems is an art of balancing between computation and communication. There have been two lines of research that try to deal with slower networks: {…