Universal sketches for the frequency negative moments and other decreasing streaming sums

Abstract

Given a stream with frequencies $f_d$, for $d\in [n]$, we characterize the space necessary for approximating the frequency negative moments $F_p=\sum |f_d{|}^p$, where $p<0$ and the sum is taken over all items $d\in [n]$ with nonzero frequency, in terms of $n$, $\epsilon$, and $m=\sum |f_d|$. To accomplish this, we actually prove a much more general result. Given any nonnegative and nonincreasing function $g$, we characterize the space necessary for any streaming algorithm that outputs a $(1\pm \epsilon )$-approximation to $\sum g(|f_d|)$, where again the sum is over items with nonzero frequency. The storage required is expressed in the form of the solution to a relatively simple nonlinear optimization problem, and the algorithm is universal for $(1\pm \epsilon )$-approximations to any such sum where the applied function is nonnegative, nonincreasing, and has the same or smaller space complexity as $g$. This partially answers an open question of Nelson (IITK Workshop Kanpur, 2009).

19 pages