RetryGuard: Preventing Self-Inflicted and Attack-Driven Retry Storms in Cloud Applications

Abstract

Modern cloud applications are built from independent microservices, offering scalability and usage-based billing. However, their reliance on independently-operating auto-scalers introduces coordination challenges. Default retry patterns can trigger "retry storms" during service miscoordination or adversarial overload, amplifying load, latency, and resource billing. These storms can cause either self-inflicted Denial-of-Wallet (DoW) or amplify the impact of DDoS attacks. To overcome these problems, we introduce RetryGuard, a distributed framework for productive control of retry patterns across interdependent microservices. By managing retry policy on a per-service basis and making parallel decisions, RetryGuard prevents retry storms, curbs resource contention, and mitigates escalating operational costs. RetryGuard makes its decisions based on an analytical model that captures the relationships among retries, throughput (rejections), delays, and costs. Simulations show that it outperforms established mechanisms, including exponential backoff, jitter, and retry budgets. Experimental results show that RetryGuard significantly reduces resource usage and costs compared to existing AWS policies, achieving more than 90% improvement in latency, and 98% reduction in storm size. We further demonstrate its effectiveness in a multi-layer Kubernetes deployment with the Istio service-mesh, where RetryGuard reduces the peak number of replicas by $3\times$ and cumulative memory usage by 55%.