Design Effect Ratios for Bayesian Survey Models: A Diagnostic Framework for Identifying Survey-Sensitive Parameters

Abstract

Bayesian hierarchical models are increasingly fitted to complex survey data through weighted pseudo-posteriors, with a post-processing step that rescales the posterior to match a design-based sandwich covariance. Applied to all parameters at once, this correction can be harmful as well as protective: parameters stabilized by hierarchical shrinkage or identified by between-group variation have their credible intervals needlessly widened or spuriously narrowed. We propose the Design Effect Ratio (DER), the ratio of a parameter's design-based sandwich variance, computed for a declared variance target, to its model-based posterior variance, as a per-parameter diagnostic for applying the correction selectively. For hierarchical Gaussian models we derive exact finite-sample expressions under explicit balance and common-design-effect hypotheses: fixed-effect DERs scale with the design effect attenuated by the between-group share of identifying variation, and random-effect DERs factor into design effect, shrinkage, and a group-count term, with a conservation identity linking the two levels. A general matrix formula covers arbitrary parameter blocks and non-Gaussian likelihoods. A simulation study with a genuine informative two-stage sampling mechanism and a re-analysis of the 2019 National Survey of Early Care and Education, reported under both the design-PSU and model-group variance targets, show the diagnostic separating survey-sensitive from shrinkage-protected parameters and the selective correction avoiding the damage of blanket rescaling. The R package svyder implements the workflow.