HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routing

Leyang Xue, Yao Fu, Luo Mai +1

Giant Deep Neural Networks (DNNs), have become indispensable for accurate and robust support of large-scale cloud based AI services. However, serving giant DNNs is prohibitively ex…