High-Throughput LLM inference on Heterogeneous Clusters

Yi Xiong, Jinqi Huang, Wenjie Huang +6

Nowadays, many companies possess various types of AI accelerators, forming heterogeneous clusters. Efficiently leveraging these clusters for high-throughput large language model (L…