SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms

Yaodan Xu, Jingzhou Sun, Sheng Zhou +1

In up-to-date machine learning (ML) applications on cloud or edge computing platforms, batching is an important technique for providing efficient and economical services at scale.…