Serving Compound Inference Systems on Datacenter GPUs

Sriram Devata, Rahul Singh, Sarita Adve

Applications in emerging domains such as XR are being built as compound inference systems, where multiple ML models are composed in the form of a task graph to service each request…