Ternary cyclotomic polynomials having a large coefficient

Abstract

Let ${\Phi }_n(x)$ denote the $n$th cyclotomic polynomial. In 1968 Sister Marion Beiter conjectured that $a_n(k)$, the coefficient of $x^k$ in ${\Phi }_n(x)$, satisfies $|a_n(k)|\le (p+1)/2$ in case $n=pqr$ with $p<q<r$ primes (in this case ${\Phi }_n(x)$ is said to be ternary). Since then several results towards establishing her conjecture have been proved (for example $|a_n(k)|\le 3p/4$). Here we show that, nevertheless, Beiter's conjecture is false for every $p\ge 11$. We also prove that given any $\epsilon >0$ there exist infinitely many triples $(p_j,q_j,r_j)$ with $p_1<p_2<...$ consecutive primes such that $|a_{p_j{q}_j{r}_j}(n_j)|>(2/3-\epsilon )p_j$ for $j\ge 1$.

19 pages, 6 tables, to appear in Crelle's Journal. Revised version with many small changes