Learning Deep Phylogeny Through Epistasis-Aware Geometry

Abstract

Epistasis shapes protein evolution because the effect of a substitution depends on its sequence background. Site-independent substitution models can recover phylogeny accurately, but their resolution declines as substitutions accumulate and reshape mutational constraints. Potts models describe family-specific epistatic landscapes but leave the histories traversed through them unresolved. We introduce a generative framework that learns family-specific evolutionary structure directly from sequences evolving on such landscapes. Our variational autoencoder uses a covariance-matching loss to target epistatic dependencies and a correlation loss to align latent distance with sequence divergence before p-distance saturates. We compare two implementations that differ only in latent geometry. EucEvoGen uses Euclidean space, and HyperEvoGen uses a Poincaré ball. Trained on natural protein alignments, both reproduce family-level sequence diversity and recover higher-order sequence statistics beyond pairwise marginals more accurately than a Potts-model generator. In simulations of sequence evolution along Yule trees under a Potts Hamiltonian, HyperEvoGen recovers the known topologies at least as accurately as maximum-likelihood inference, improving on it for four of five tree-distance metrics. Its ancestral sequence reconstructions trail maximum likelihood for closely related sequences, overtake it beyond roughly 0.3 p-distance, and retain that advantage as divergence increases. EucEvoGen matches HyperEvoGen in sequence generation but yields less accurate phylogenetic inference. This comparison links the accuracy of deep phylogenetic inference to the global organization of the latent space. These results suggest that family-specific representations of epistatically constrained sequence space can extend phylogenetic inference into the regime where stationary site-independent models lose resolution.

arXiv comment28 pages, 7 figures, 1 Table