Subgroup Inconsistency and the Dilution Effect in Levene's Test for Homoscedasticity

Abstract

Levene's test for homoscedasticity is a standard procedure used to evaluate whether multiple groups of independent observations share a common variance. Because Levene's test relies on an Analysis of Variance (ANOVA) applied to transformed absolute or squared deviations, it structurally mirrors the statistical properties of ANOVA itself. Let ${\mu }_j$ and ${\sigma }_j^2$ denote the expectation and variance of the observations in group $j$. It was previously established that for a given significance level $\alpha$, ANOVA can result in a logical contradiction: failing to reject the global null hypothesis $H_0:{\mu }_1={\mu }_2={\mu }_3$ while simultaneously rejecting the localized hypothesis $H_0^{\prime }:{\mu }_1={\mu }_2$ with the same or higher confidence. In this paper, we show that Levene's test directly inherits this same ``paradox'' regarding group variances ${\sigma }_j^2$. We provide theoretical reasoning and a numerical illustration of this inconsistency, demonstrating how the addition of a well-behaved third group can dilute the test statistic and mask a significant localized variance discrepancy.