57 citations · 253 across the 32 of their papers we have counts for
11 papers · 1 filter
Accelerating Cross-Model LLM Inference via Attention Map Reuse
Zhan Jin, Yuze Luo, Bin Cui +1
LLM-augmented data workflows often involve multiple heterogeneous models operating over substantial shared context, such as schemas, tool specifications, interaction history, and i…
DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning
Yujie Wang, Siwei Chen, Longzan Luo +4
Reinforcement Learning (RL) has become pivotal for improving model capabilities yet suffers from rollout efficiency bottlenecks due to the long-tail response length distribution. W…
Experimental Analysis of Large-scale Learnable Vector Storage Compression
Hailin Zhang, Penghao Zhao, Xupeng Miao +4
Learnable embedding vector is one of the most important applications in machine learning, and is widely used in various database-related domains. However, the high dimensionality o…
Improving Automatic Parallel Training via Balanced Memory Workload Optimization
Yujie Wang, Youhe Jiang, Xupeng Miao +5
Transformer models have emerged as the leading approach for achieving state-of-the-art performance across various application domains, serving as the foundation for advanced large-…
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
Xiaonan Nie, Yi Liu, Fangcheng Fu +5
Recent years have witnessed the unprecedented achievements of large-scale pre-trained models, especially the Transformer models. Many products and services in Tencent Inc., such as…
Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
Xupeng Miao, Yujie Wang, Youhe Jiang +4
Transformer models have achieved state-of-the-art performance on various domains of applications and gradually becomes the foundations of the advanced large deep learning (DL) mode…
Distributed Graph Neural Network Training: A Survey
Yingxia Shao, Hongzheng Li, Xizhi Gu +6
Graph neural networks (GNNs) are a type of deep learning models that are trained on graphs and have been successfully applied in various domains. Despite the effectiveness of GNNs,…
Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updates
Fangcheng Fu, Xupeng Miao, Jiawei Jiang +2
Vertical federated learning (VFL) is an emerging paradigm that allows different parties (e.g., organizations or enterprises) to collaboratively build machine learning models with p…
HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework
Xupeng Miao, Hailin Zhang, Yining Shi +4
Embedding models have been an effective learning paradigm for high-dimensional data. However, one open issue of embedding models is that their representations (latent factors) ofte…
EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate
Xiaonan Nie, Xupeng Miao, Shijie Cao +7
Mixture-of-experts (MoE) is becoming popular due to its success in improving the model quality, especially in Transformers. By routing tokens with a sparse gate to a few experts (i…
ROD: Reception-aware Online Distillation for Sparse Graphs
Wentao Zhang, Yuezihan Jiang, Yang Li +6
Graph neural networks (GNNs) have been widely used in many graph-based tasks such as node classification, link prediction, and node clustering. However, GNNs gain their performance…