Whittaker supports for representations of reductive groups

Abstract

Let $F$ be either $\mathbb{R}$ or a finite extension of ${\mathbb{Q}}_p$, and let $G$ be a finite central extension of the group of $F$-points of a reductive group defined over $F$. Also let $\pi$ be a smooth representation of $G$ (Frechet of moderate growth if $F=\mathbb{R}$). For each nilpotent orbit $\mathcal{O}$ we consider a certain Whittaker quotient ${\pi }_{\mathcal{O}}$ of $\pi$. We define the Whittaker support WS$(\pi )$ to be the set of maximal $\mathcal{O}$ among those for which ${\pi }_{\mathcal{O}}\ne 0$. In this paper we prove that all $\mathcal{O}\in \mathrm{WS}(\pi )$ are quasi-admissible nilpotent orbits, generalizing some of the results in [Moe96,JLS16]. If $F$ is $p$-adic and $\pi$ is quasi-cuspidal then we show that all $\mathcal{O}\in \mathrm{WS}(\pi )$ are $F$-distinguished, i.e. do not intersect the Lie algebra of any proper Levi subgroup of $G$ defined over $F$. We also give an adaptation of our argument to automorphic representations, generalizing some results from [GRS03,Shen16,JLS16,Cai] and confirming some conjectures from [Ginz06]. Our methods are a synergy of the methods of the above-mentioned papers, and of our preceding paper [GGS17].

v7: minor corrections. Version to appear in Annales de l'institut Fourier. 33 pages