Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models

Abstract

For deep learning, size is power. Massive neural nets trained on broad data for a spectrum of tasks are at the forefront of artificial intelligence. These large pre-trained models or Jacks of All Trades (JATs), when fine-tuned for downstream tasks, are gaining importance in driving deep learning advancements. However, environments with tight resource constraints, changing objectives and intentions, or varied task requirements, could limit the real-world utility of a singular JAT. Hence, in tandem with current trends towards building increasingly large JATs, this paper conducts an initial exploration into concepts underlying the creation of a diverse set of compact machine learning model sets. Composed of many smaller and specialized models, the Set of Sets is formulated to simultaneously fulfil many task settings and environmental conditions. A means to arrive at such a set tractably in one pass of a neuroevolutionary multitasking algorithm is presented for the first time, bringing us closer to models that are collectively Masters of All Trades.

This article has been published in the IEEE Computational Intelligence Magazine. We have updated the article's Title, Authors, and Abstract in the latest version

References in corpus (7)

• On the Opportunities and Risks of Foundation Models
• Scaling Laws for Neural Language Models
• A Survey on Evolutionary Neural Architecture Search
• Beyond English-Centric Multilingual Machine Translation
• A Review of Evolutionary Multi-modal Multi-objective Optimization
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Learning to Branch for Multi-Task Learning

Cited by in corpus (1)

• Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities