Efficiency requires innovation

Abstract

In estimation a parameter $\theta \in \mathbb{R}$ from a sample $(x_1,\dots ,x_n)$ from a population $P_{\theta }$ a simple way of incorporating a new observation $x_{n+1}$ into an estimator ${\tilde{\theta }}_n={\tilde{\theta }}_n(x_1,\dots ,x_n)$ is transforming ${\tilde{\theta }}_n$ to what we call the {\it jackknife extension} ${\tilde{\theta }}_{n+1}^{(e)}={\tilde{\theta }}_{n+1}^{(e)}(x_1,\dots ,x_n,x_{n+1})$, \[\tildeθ_{n+1}^{(e)} = \{\tildeθ_n (x_1 ,\ldots,x_n)+ \tildeθ_n (x_{n+1},x_2 ,\ldots,x_n) + \ldots + \tildeθ_n (x_1 ,\ldots,x_{n-1},x_{n+1})\}/(n+1).\] Though ${\tilde{\theta }}_{n+1}^{(e)}$ lacks an innovation the statistician could expect from a larger data set, it is still better than ${\tilde{\theta }}_n$, \[{\rm var}(\tildeθ_{n+1}^{(e)})\leq\frac{n}{n+1} {\rm var}(\tildeθ_n).\] However, an estimator obtained by jackknife extension for all $n$ is asymptotically efficient only for samples from exponential families. For a general $P_{\theta }$, asymptotically efficient estimators require innovation when a new observation is added to the data. Some examples illustrate the concept.