On Data-centric Myths
arXiv:2111.11514
Abstract
The community lacks theory-informed guidelines for building good data sets. We analyse theoretical directions relating to what aspects of the data matter and conclude that the intuitions derived from the existing literature are incorrect and misleading. Using empirical counter-examples, we show that 1) data dimension should not necessarily be minimised and 2) when manipulating data, preserving the distribution is inessential. This calls for a more data-aware theoretical understanding. Although not explored in this work, we propose the study of the impact of data modification on learned representations as a promising research direction.
arXiv admin note: text overlap with arXiv:2110.13968
References in corpus (9)
- mixup: Beyond Empirical Risk Minimization
- Estimating the intrinsic dimension of datasets by a minimal neighborhood information
- Data Augmentation by Pairing Samples for Images Classification
- Intrinsic dimension of data representations in deep neural networks
- Deep learning generalizes because the parameter-function map is biased towards simple functions
- Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
- Bayesian Distance Clustering
- On Mixup Regularization
- Distributional Results for Model-Based Intrinsic Dimension Estimators