WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

Fanzhe Wei, Li Liu, Ziyang Wang +1

KV-cache quantization is validated today by offline benchmark averages; a deployed system cannot tell whether compression is damaging the request it is serving right now. We give i…