The potential and perils of preprocessing: Building new foundations
arXiv:1309.6790 · doi:10.3150/13-BEJSP16
Abstract
Preprocessing forms an oft-neglected foundation for a wide range of statistical and scientific analyses. However, it is rife with subtleties and pitfalls. Decisions made in preprocessing constrain all later analyses and are typically irreversible. Hence, data analysis becomes a collaborative endeavor by all parties involved in data collection, preprocessing and curation, and downstream inference. Even if each party has done its best given the information and resources available to them, the final result may still fall short of the best possible in the traditional single-phase inference framework. This is particularly relevant as we enter the era of "big data". The technologies driving this data explosion are subject to complex new forms of measurement error. Simultaneously, we are accumulating increasingly massive databases of scientific analyses. As a result, preprocessing has become more vital (and potentially more dangerous) than ever before.
arXiv commentPublished in at http://dx.doi.org/10.3150/13-BEJSP16 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)
References in corpus (8)
- In silico prediction of protein-protein interactions in human macrophages
- The Effect of Line of Sight Temperature Variation and Noise on Dust Continuum Observations
- Inverse temperature dependence of the dust submillimeter spectral index
- Submillimetre point sources from the Archeops experiment: Very Cold Clumps in the Galactic Plane
- Dust SEDs in the era of Herschel and Planck: a Hierarchical Bayesian fitting technique
- Variations of the spectral index of dust emissivity from Hi-GAL observations of the Galactic plane
- The physical properties of the dust in the RCW 120 HII region as seen by Herschel
- On surrogate loss functions and -divergences