OTAS: An Elastic Transformer Serving System via Token Adaptation

Jinyu Chen, Wenchao Xu, Zicong Hong +4

Transformer model empowered architectures have become a pillar of cloud services that keeps reshaping our society. However, the dynamic query loads and heterogeneous user requireme…