Commutators of small rank and reducibility of operator semigroups

Abstract

It is easy to see that if $\cG$ is a non-abelian group of unitary matrices, then for no members $A$ and $B$ of $\cG$ can the rank of $AB-BA$ be one. We examine the consequences of the assumption that this rank is at most two for a general semigroup $\cS$ of linear operators. Our conclusion is that under obviously necessary, but trivial, size conditions, $\cS$ is reducible. In the case of a unitary group satisfying the hypothesis, we show that it is contained in the direct sum $\cG_1\oplus\cG_2$ where $\cG_1$ is at most $3\times 3$ and $\cG_2$ is abelian.

To appear in Proceeding of the American Mathematical Society