Limit theorems for the site frequency spectrum of neutral mutations in an exponentially growing population

Abstract

The site frequency spectrum (SFS) is a widely used summary statistic of genomic data. Motivated by recent evidence for the role of neutral evolution in cancer, we investigate the SFS of neutral mutations in an exponentially growing population. Using branching process techniques, we establish (first-order) almost sure convergence results for the SFS of a Galton-Watson process, evaluated either at a fixed time or at the stochastic time at which the population first reaches a certain size. We finally use our results to construct consistent estimators for the extinction probability and the effective mutation rate of a birth-death process.

35 pages. Main changes from v1: (1) Fixed-size result improved to almost sure convergence, (2) error bound in Lemma 1 and handling of error terms in its proof improved