293 citations · 499 across the 27 of their papers we have counts for
27 papers
SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity
Zhenghao Gan, Yichen Bao, Yifei Liu +3
Efficient LLM inference scheduling is crucial for user experience. However, LLM inferences exhibit remarkable demand uncertainty (with unknown output length beforehand) and hybridi…
Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management
Hang Zhang, Jiuchen Shi, Yixiao Wang +3
Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA ad…
A Microservice Graph Generator with Production Characteristics
Fanrong Du, Jiuchen Shi, Quan Chen +2
A production microservice application may provide multiple services, queries of a service may have different call graphs, and a microservice may be shared across call graphs. It is…
Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design
Chunyu Xue, Weihao Cui, Quan Chen +10
Efficiently training large-scale models (LMs) in GPU clusters involves two separate avenues: inter-job dynamic scheduling and intra-job adaptive parallelism (AP). However, existing…
Accelerating Sparse DNNs Based on Tiled GEMM
Cong Guo, Fengchen Xue, Jingwen Leng +5
Network pruning can reduce the computation cost of deep neural network (DNN) models. However, sparse models often produce randomly-distributed weights to maintain accuracy, leading…
An Optimizing Framework on MLIR for Efficient FPGA-based Accelerator Generation
Weichuang Zhang, Jieru Zhao, Guan Shen +3
With the increasing demand for computing capability given limited resource and power budgets, it is crucial to deploy applications to customized accelerators like FPGAs. However, F…
JUNO: Optimizing High-Dimensional Approximate Nearest Neighbour Search with Sparsity-Aware Algorithm and Ray-Tracing Core Mapping
Zihan Liu, Wentao Ni, Jingwen Leng +6
Approximate nearest neighbor (ANN) search is a widely applied technique in modern intelligent applications, such as recommendation systems and vector databases. Therefore, efficien…
Adaptive CPU Resource Allocation for Emulator in Kernel-based Virtual Machine
Yecheng Yang, Pu Pang, Jiawen Wang +2
The technologies of heterogeneous multi-core architectures, co-location, and virtualization can be used to reduce server power consumption and improve system utilization, which are…
STAG: Enabling Low Latency and Low Staleness of GNN-based Services with Dynamic Graphs
Jiawen Wang, Quan Chen, Deze Zeng +3
Many emerging user-facing services adopt Graph Neural Networks (GNNs) to improve serving accuracy. When the graph used by a GNN model changes, representations (embedding) of nodes…
MARS: Exploiting Multi-Level Parallelism for DNN Workloads on Adaptive Multi-Accelerator Systems
Guan Shen, Jieru Zhao, Zeke Wang +5
Along with the fast evolution of deep neural networks, the hardware system is also developing rapidly. As a promising solution achieving high scalability and low manufacturing cost…
DFlow: Efficient Dataflow-based Invocation Workflow Execution for Function-as-a-Service
Xiaoxiang Shi, Chao Li, Zijun Li +5
The Serverless Computing is becoming increasingly popular due to its ease of use and fine-grained billing. These features make it appealing for stateful application or serverless w…
AdaptGear: Accelerating GNN Training via Adaptive Subgraph-Level Kernels on GPUs
Yangjie Zhou, Yaoxu Song, Jingwen Leng +7
Graph neural networks (GNNs) are powerful tools for exploring and learning from graph structures and features. As such, achieving high-performance execution for GNNs becomes crucia…
DataFlower: Exploiting the Data-flow Paradigm for Serverless Workflow Orchestration
Zijun Li, Chuhao Xu, Quan Chen +3
Serverless computing that runs functions with auto-scaling is a popular task execution pattern in the cloud-native era. By connecting serverless functions into workflows, tenants c…
Async-fork: Mitigating Query Latency Spikes Incurred by the Fork-based Snapshot Mechanism from the OS Level
Pu Pang, Gang Deng, Kaihao Bai +12
In-memory key-value stores (IMKVSes) serve many online applications because of their efficiency. To support data backup, popular industrial IMKVSes periodically take a point-in-tim…
SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences
Guan Shen, Jieru Zhao, Quan Chen +3
The attention mechanisms of transformers effectively extract pertinent information from the input sequence. However, the quadratic complexity of self-attention w.r.t the sequence l…
VELTAIR: Towards High-Performance Multi-tenant Deep Learning Services via Adaptive Compilation and Scheduling
Zihan Liu, Jingwen Leng, Zhihui Zhang +3
Deep learning (DL) models have achieved great success in many application domains. As such, many industrial companies such as Google and Facebook have acknowledged the importance o…
The Serverless Computing Survey: A Technical Primer for Design Architecture
Zijun Li, Linsong Guo, Jiagan Cheng +3
The development of cloud infrastructures inspires the emergence of cloud-native computing. As the most promising architecture for deploying microservices, serverless computing has…
Characterizing and Demystifying the Implicit Convolution Algorithm on Commercial Matrix-Multiplication Accelerators
Yangjie Zhou, Mengtian Yang, Cong Guo +5
Many of today's deep neural network accelerators, e.g., Google's TPU and NVIDIA's tensor core, are built around accelerating the general matrix multiplication (i.e., GEMM). However…
Dubhe: Towards Data Unbiasedness with Homomorphic Encryption in Federated Learning Client Selection
Shulai Zhang, Zirui Li, Quan Chen +3
Federated learning (FL) is a distributed machine learning paradigm that allows clients to collaboratively train a model over their own local data. FL promises the privacy of client…
Pagurus: Eliminating Cold Startup in Serverless Computing with Inter-Action Container Sharing
Zijun Li, Quan Chen, Minyi Guo
Serverless computing provides fine-grain resource sharing between Cloud tenants through containers. Each function invocation (action) runs in an individual container. When there is…
DLFusion: An Auto-Tuning Compiler for Layer Fusion on Deep Neural Network Accelerator
Zihan Liu, Jingwen Leng, Quan Chen +4
Many hardware vendors have introduced specialized deep neural networks (DNN) accelerators owing to their superior performance and efficiency. As such, how to generate and optimize…
How Far Does BERT Look At:Distance-based Clustering and Analysis of BERTs Attention
Yue Guan, Jingwen Leng, Chao Li +2
Recent research on the multi-head attention mechanism, especially that in pre-trained models such as BERT, has shown us heuristics and clues in analyzing various aspects of the mec…
Towards QoS-Aware and Resource-Efficient GPU Microservices Based on Spatial Multitasking GPUs In Datacenters
Wei Zhang, Quan Chen, Kaihua Fu +6
While prior researches focus on CPU-based microservices, they are not applicable for GPU-based microservices due to the different contention patterns. It is challenging to optimize…
Balancing Efficiency and Flexibility for DNN Acceleration via Temporal GPU-Systolic Array Integration
Cong Guo, Yangjie Zhou, Jingwen Leng +6
The research interest in specialized hardware accelerators for deep neural networks (DNN) spikes recently owing to their superior performance and efficiency. However, today's DNN a…