Extremal rays of the embedded subgroup saturation cone

Abstract

We examine the extremal rays of the cone of dominant weights $(\mu ,\hat{\mu })$ for groups $G\subseteq \hat{G}$ for which there exists $N\gg 0$ such that $${\left(V(N\mu )\otimes V(N\hat{\mu })\right)}^G\ne (0).$$ We exhibit formulas for a class of rays ("type I") on any regular face of the cone. These rays are identified thanks to a generalization of Fulton's conjecture, which we prove along the way. We verify that the remaining rays ("type II") on the face are the images of extremal rays for a smaller cone under a certain map, whose formula is given. A procedure is given for finding the rays of the cone not on any regular face. This is a generalization of the work of Belkale and Kiers on extremal rays for the saturated tensor cone; the specialization is given by $\hat{G}=G\times G$ with the diagonal embedding of $G$. We include several examples to illustrate the formulas.

50 pages, substantial edits to accommodate $G$ reductive instead of semisimple, new examples. To appear in Annales de l'Institut Fourier