Dimension estimates for the set of points with non-dense orbit in homogeneous spaces

Abstract

Let $X=G/\Gamma$, where $G$ is a Lie group and $\Gamma$ is a lattice in $G$, and let $U$ be a subset of $X$ whose complement is compact. We use the exponential mixing results for diagonalizable flows on $X$ to give upper estimates for the Hausdorff dimension of the set of points whose trajectories miss $U$. This extends a recent result of Kadyrov and produces new applications to Diophantine approximation, such as an upper bound for the Hausdorff dimension of the set of weighted uniformly badly approximable systems of linear forms, generalizing an estimate due to Broderick and Kleinbock.

26 pages; more explanations added and several misprints corrected