NExt-LMM: a two-stage LMM framework with near-exact variance components for genome-wide association studies

Abstract

Linear mixed models (LMMs) are widely used in genome-wide association studies (GWAS) to account for population stratification and cryptic relatedness. However, their practical application remains computationally challenging because parameter estimation and association testing require large-scale operations on genetic similarity matrices (GSMs). Here, we present NExt-LMM, a two-stage framework for scalable LMM-based GWAS. In the first stage, NExt-LMM obtains a preliminary estimate of a shared variance ratio under the null model. In the second stage, it constructs the corresponding covariance matrix and accelerates downstream inference through a Hierarchical Off-Diagonal Low-Rank (HODLR) representation. Specifically, NExt-LMM exploits the low-rank structure of off-diagonal covariance blocks to enable substantially faster matrix factorization and inverse-related computations than conventional dense decompositions. It then reuses the resulting HODLR-based covariance representation to accelerate maximum-likelihood-based inference under the shared variance-ratio setting. We further establish error bounds that account for both sources of approximation in the framework: the preliminary variance-ratio estimate and the HODLR approximation. In particular, we show that, conditional on a fixed variance ratio, the NExt-LMM estimator can be made arbitrarily close to the exact shared-variance estimator as the matrix approximation tolerance approaches zero. \AAA{Numerical experiments show that NExt-LMM improves computational efficiency while maintaining high accuracy relative to existing methods, and that the size of this improvement is governed by the numerical rank retained at the tolerance required for calibrated association inference.} We provide an open-source Python implementation at https://github.com/ZhibinPU/NExt-LMM.

arXiv comment36 pages, 13 figures