No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions

Benjamin S. Ruben, William L. Tong, Hamza Tahir Chaudhry +1

Given a fixed budget for total model size, one must choose between training a single large model or combining the predictions of multiple smaller models. We investigate this trade-…