Computing nonsurjective primes associated to Galois representations of genus $2$ curves

Abstract

For a genus $2$ curve $C$ over $\mathbb{Q}$ whose Jacobian $A$ admits only trivial geometric endomorphisms, Serre's open image theorem for abelian surfaces asserts that there are only finitely many primes $\ell$ for which the Galois action on $\ell$-torsion points of $A$ is not maximal. Building on work of Dieulefait, we give a practical algorithm to compute this finite set. The key inputs are Mitchell's classification of maximal subgroups of $\mathrm{PS}{\mathrm{p}}_4({\mathbb{F}}_{\ell })$, sampling of the characteristic polynomials of Frobenius, and the Khare--Wintenberger modularity theorem. The algorithm has been submitted for integration into Sage, executed on all of the genus~$2$ curves with trivial endomorphism ring in the LMFDB, and the results incorporated into the homepage of each such curve.