Orthogonality of invariant vectors

Abstract

Let $G$ be a finite group with given subgroups $H$ and $K$. Let $\pi$ be an irreducible complex representation of $G$ such that its space of $H$-invariant vectors as well as the space of $K$-invariant vectors are both one dimensional. Let $v_H$ (resp. $v_K$) denote an $H$-invariant (resp. $K$-invariant) vector of unit norm in the standard $G$-invariant inner product $⟨,{⟩}_{\pi }$ on $\pi$. Our interest is in computing the square of the absolute value of $⟨v_H,v_K{⟩}_{\pi }$. This is the correlation constant $c(\pi ;H,K)$ defined by Gross. In this paper, we give a sufficient condition for $⟨v_H,v_K{⟩}_{\pi }$ to be zero and a sufficient condition for it to be non-zero (i.e., $H$ and $K$ are correlated with respect to $\pi$), when $G={\mathrm{GL}}_2({\mathbb{F}}_q)$, where ${\mathbb{F}}_q$ is the finite field of $q=p^f$ elements of odd characteristic $p$, $H$ is its split torus and $K$ is a non-split torus. The key idea in our proof is to analyse the mod $p$ reduction of $\pi$. We give an explicit formula for $|⟨v_H,v_K{⟩}_{\pi }{|}^2$ modulo $p$. Finally, we study the behaviour of $⟨v_H,v_K{⟩}_{\pi }$ under the Shintani base change and give a sufficient condition for $⟨v_H,v_K{⟩}_{\pi }$ to vanish for an irreducible representation $\pi =\mathrm{BC}(\tau )$ of ${\mathrm{PGL}}_2(\mathbb{E})$, in terms of the epsilon factor of the base changing representation $\tau$ of ${\mathrm{PGL}}_2(\mathbb{F})$, where $\mathbb{E}/\mathbb{F}$ is a finite extension of finite fields. This is reminiscent of the vanishing of $L(1/2,\mathrm{BC}(\tau ))$, in the theory of automorphic forms, when the global root number of $\tau$ is $-1$.