Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features

Abstract

Randomized algorithms exploit stochasticity to reduce computational complexity. One important example is random feature regression (RFR) that accelerates Gaussian process regression (GPR). RFR approximates an unknown function with a random neural network whose hidden weights and biases are sampled from a probability distribution. Only the final output layer is fit to data. In randomized algorithms like RFR, the hyperparameters that characterize the sampling distribution greatly impact performance, yet are not directly accessible from samples. This makes optimization of hyperparameters via standard (gradient-based) optimization tools inapplicable. Inspired by Bayesian ideas from GPR, this paper introduces a random objective function that is tailored for hyperparameter tuning of vector-valued random features. The objective is minimized with ensemble Kalman inversion (EKI). EKI is a gradient-free particle-based optimizer that is scalable to high-dimensions and robust to randomness in objective functions. A numerical study showcases the new black-box methodology to learn hyperparameter distributions in several problems that are sensitive to the hyperparameter selection: two global sensitivity analyses, integrating a chaotic dynamical system, and solving a Bayesian inverse problem from atmospheric dynamics. The success of the proposed EKI-based algorithm for RFR suggests its potential for automated optimization of hyperparameters arising in other randomized algorithms.

References in corpus (13)

• Next Generation Reservoir Computing
• The prior can generally only be understood in the context of the likelihood
• The Ensemble Kalman Filter for Inverse Problems
• Quantifying and Reducing Model-Form Uncertainties in Reynolds-Averaged Navier-Stokes Simulations: A Data-Driven, Physics-Based Bayesian Approach
• On Explaining the Surprising Success of Reservoir Computing Forecaster of Chaos? The Universal Machine Learning Dynamical System with Contrasts to VAR and DMD
• Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM
• Forecasting Chaotic Systems with Very Low Connectivity Reservoir Computers
• Calibrate, Emulate, Sample
• Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks
• The Random Walk Metropolis: Linking Theory and Practice Through a Case Study
• Kernel Flows: from learning kernels from data into the abyss
• Calibration and Uncertainty Quantification of Convective Parameters in an Idealized GCM
• Explainable Offline-Online Training of Neural Networks for Parameterizations: A 1D Gravity Wave-QBO Testbed in the Small-data Regime