33 citations · 68 across the 15 of their papers we have counts for
17 papers
Transforming the Use of Earth Observation Data: Exascale Training of a Generative Compression Model with Historical Priors for up to 10,000x Data Reduction
Jinxiao Zhang, Runmin Dong, Xiyong Wu +14
Earth observation is becoming one of the largest data-producing activities in science, yet current pipelines still treat compression as a storage and transmission tool rather than…
DiT-HC: Enabling Efficient Training of Visual Generation Model DiT on HPC-oriented CPU Cluster
Jinxiao Zhang, Yunpu Xu, Xiyong Wu +7
Generative foundation models have become an important tool for data reconstruction and simulation in scientific computing, showing a tight integration with traditional numerical si…
Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving
Ziming Liu, Boyu Tian, Guoteng Wang +15
Mixture-of-Experts (MoE) models challenge serving infrastructures with dynamic, sparse expert utilization, causing instability on conventional systems designed for dense architectu…
Adaptive Hybrid Caching for Efficient Text-to-Video Diffusion Model Acceleration
Yuanxin Wei, Lansong Diao, Bujiao Chen +6
Efficient video generation models are increasingly vital for multimedia synthetic content generation. Leveraging the Transformer architecture and the diffusion process, video DiT m…
ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
Zedong Liu, Shenggan Cheng, Guangming Tan +2
Multimodal large language models (MLLMs) extend LLMs to handle images, videos, and audio by incorporating feature extractors and projection modules. However, these additional compo…
HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelism
Geng Zhang, Shenggan Cheng, Xuanlei Zhao +2
As transformer sequence lengths grow, existing pipeline parallelisms incur suboptimal performance due to the quadratic attention computation and the substantial memory overhead. To…
SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperation
Shenggan Cheng, Yuanxin Wei, Lansong Diao +8
Leveraging the diffusion transformer (DiT) architecture, models like Sora, CogVideoX and Wan have achieved remarkable progress in text-to-video, image-to-video, and video editing t…
StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training
Ziming Liu, Shaoyu Wang, Shenggan Cheng +5
Training Transformer models on long sequences in a distributed setting poses significant challenges in terms of efficiency and scalability. Current methods are either constrained b…
HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices
Xuanlei Zhao, Bin Jia, Haotian Zhou +3
In recent times, the emergence of Large Language Models (LLMs) has resulted in increasingly larger model size, posing challenges for inference on low-resource devices. Prior approa…
DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Xuanlei Zhao, Shenggan Cheng, Chang Chen +4
Scaling multi-dimensional transformers to long sequences is indispensable across various domains. However, the challenges of large memory requirements and slow speeds of such seque…
AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference
Xuanlei Zhao, Shenggan Cheng, Guangyang Lu +5
Large deep learning models have achieved impressive performance across a range of applications. However, their large memory requirements, including parameter memory and activation…
Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
Ziming Liu, Shenggan Cheng, Haotian Zhou +1
Large-scale language models have become increasingly challenging and expensive to train. Among various methods addressing this issue, Pipeline Parallelism has been widely employed…
ATP: Adaptive Tensor Parallelism for Foundation Models
Shenggan Cheng, Ziming Liu, Jiangsu Du +1
Foundation models have impressive performance and generalization capabilities across a wide range of applications. The increasing size of the models introduces great challenges for…
FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours
Shenggan Cheng, Xuanlei Zhao, Guangyang Lu +7
Protein structure prediction helps to understand gene translation and protein function, which is of growing interest and importance in structural biology. The AlphaFold model, whic…
tcFFT: Accelerating Half-Precision FFT through Tensor Cores
Binrui Li, Shenggan Cheng, James Lin
Fast Fourier Transform (FFT) is an essential tool in scientific and engineering computation. The increasing demand for mixed-precision FFT has made it possible to utilize half-prec…
CUBE -- Towards an Optimal Scaling of Cosmological N-body Simulations
Shenggan Cheng, Hao-Ran Yu, Derek Inman +3
N-body simulations are essential tools in physical cosmology to understand the large-scale structure (LSS) formation of the Universe. Large-scale simulations with high resolution a…
HMS-Net: Hierarchical Multi-scale Sparsity-invariant Network for Sparse Depth Completion
Zixuan Huang, Junming Fan, Shenggan Cheng +3
Dense depth cues are important and have wide applications in various computer vision tasks. In autonomous driving, LIDAR sensors are adopted to acquire depth measurements around th…