Differential Refresh of Materialized Derived State: From a Single-Age Equivalence Limit to an Optimal Per-Partition Allocation, with a Real-System Validation on Materialized Views

Abstract

Materialized views, cached aggregates, and trained models are all derived state: query results precomputed from base relations that begin to age the instant they are built, because the data they summarize keeps changing. A common way to keep such state fresh is to replace a fixed recompute cadence with an adaptive trigger -- a weighted staleness score that recomputes when accumulated base-data change crosses a threshold. We show this is the wrong lever and identify the right one. First, an equivalence limit: any refresh trigger that is a static, strictly monotone function of a single shared age of a globally-recomputed artifact is operationally equivalent to a calibrated periodic timer, so a global weighted staleness budget cannot schedule a recompute that a uniform timer could not. The escape is to enlarge the state: refresh partitions (the groups of a GROUP BY, per-tenant views, or data segments) differentially, each on its own schedule. We solve the resulting budget-allocation problem: the optimal per-partition recompute rate is proportional to the square root of partition risk sqrt(w_j*lambda_j), and the policy never costs more than the uniform timer, beating it by a closed-form Cauchy-Schwarz "price of uniformity" that grows with heterogeneity. We validate three ways -- analytic, discrete-event simulation, and a real relational engine (SQLite) maintaining materialized GROUP BY views with real SQL and wall-clock timing -- where the policy lowers realized weighted staleness by 14-15% at matched recompute count, winning on every run, while a naive per-partition exposure threshold does not reliably help. The rule for any periodically-recomputed derived state: a global staleness score is a timer in disguise; the leverage is differential refresh allocated by the square root of partition risk.