Scheduling the Unschedulable: Taming Black-Box LLM Inference at Scale

Renzhong Yuan, Yijun Zeng, Xiaosong Gao +3

When output token counts can be predicted at submission time (Gan et al., 2026), client-side scheduling against a black-box LLM API becomes semi-clairvoyant: decisions condition on…