Cardinalities of Jordan Diophantine sets of upper triangular integer matrices

Abstract

Let $U$ be the ring of upper triangular $2\times 2$ integer matrices. For $N\in U$, a Jordan $D(N)$-set is a set of distinct nonzero matrices in $U$ such that $(AB+BA)/2+N$ is a square in $U$ for any two distinct elements $A$ and $B$. We determine all $N$ for which an infinite Jordan $D(N)$-set exists. If no infinite set exists, every such set has at most six elements. If $N$ is not a difference of two squares in $U$, the bound is five. Both bounds are best possible. We also give necessary and sufficient conditions for the existence of infinite sets with nonzero pairwise Jordan products and of infinite sets consisting of nonsingular matrices.