Should attention be all we need? The epistemic and ethical implications of unification in machine learning
arXiv:2205.08377 · doi:10.1145/3531146.3533206
Abstract
"Attention is all you need" has become a fundamental precept in machine learning research. Originally designed for machine translation, transformers and the attention mechanisms that underpin them now find success across many problem domains. With the apparent domain-agnostic success of transformers, many researchers are excited that similar model architectures can be successfully deployed across diverse applications in vision, language and beyond. We consider the benefits and risks of these waves of unification on both epistemic and ethical fronts. On the epistemic side, we argue that many of the arguments in favor of unification in the natural sciences fail to transfer over to the machine learning case, or transfer over only under assumptions that might not hold. Unification also introduces epistemic risks related to portability, path dependency, methodological diversity, and increased black-boxing. On the ethical side, we discuss risks emerging from epistemic concerns, further marginalizing underrepresented perspectives, the centralization of power, and having fewer models across more domains of application
arXiv commentForthcoming in FAccT 2022
References in corpus (11)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- On the Opportunities and Risks of Foundation Models
- The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence
- Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
- A ConvNet for the 2020s
- Algorithmic Monoculture and Social Welfare
- In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
- Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity
- A Short-term Intervention for Long-term Fairness in the Labor Market
- Pretrained AI Models: Performativity, Mobility, and Change
- Can Single Neurons Solve MNIST? The Computational Power of Biological Dendritic Trees