Structure of connected nested automorphism groups

Abstract

In this article, we describe the maximal unipotent subgroups of $\mathrm{Aut}(X)$, where $X$ is an affine algebraic variety. Every subgroup of this type has a structure analogous to that of the group of triangular automorphisms of ${\mathbb{A}}^n$. In particular, it is nested, that is, a countable increasing union of algebraic subgroups. We show that a subgroup $G\subset \mathrm{Aut}(X)$ consisting of unipotent elements is closed if and only if it is nested. This implies that a connected nested subgroup of $\mathrm{Aut}(X)$ is closed, thus answering a question posed by Kraft and Zaidenberg (2022). We also extend the recent description of maximal commutative unipotent subgroups of $\mathrm{Aut}(X)$ due to Regeta and van Santen (2024), by providing a direct construction of such subgroups within our approach.

18 pages; the contents rearranged and trimmed, key proofs shortened, auxiliary results omitted, a result on abstract unipotent subgroups weakened