415 citations · 725 across the 47 of their papers we have counts for
12 papers · 1 filter
Efficient Activation Function Optimization through Surrogate Modeling
Garrett Bingham, Risto Miikkulainen
Carefully designed activation functions can improve the performance of neural networks in many machine learning tasks. However, it is difficult for humans to construct optimal acti…
AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks
Garrett Bingham, Risto Miikkulainen
Neural networks require careful weight initialization to prevent signals from exploding or vanishing. Existing initialization schemes solve this problem in specific cases by assumi…
Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model
Xin Qiu, Risto Miikkulainen
As neural network classifiers are deployed in real-world applications, it is crucial that their failures can be detected reliably. One practical solution is to assign confidence sc…
Discovering Parametric Activation Functions
Garrett Bingham, Risto Miikkulainen
Recent studies have shown that the choice of activation function can significantly affect the performance of deep learning networks. However, the benefits of novel activation funct…
Evolutionary Optimization of Deep Learning Activation Functions
Garrett Bingham, William Macke, Risto Miikkulainen
The choice of activation function can have a large effect on the performance of a neural network. While there have been some attempts to hand-engineer novel activation functions, t…
Improving Neural Network Learning Through Dual Variable Learning Rates
Elizabeth Liner, Risto Miikkulainen
This paper introduces and evaluates a novel training method for neural networks: Dual Variable Learning Rates (DVLR). Building on insights from behavioral psychology, the dual lear…
Optimizing Loss Functions Through Multivariate Taylor Polynomial Parameterization
Santiago Gonzalez, Risto Miikkulainen
Metalearning of deep neural network (DNN) architectures and hyperparameters has become an increasingly important area of research. Loss functions are a type of metaknowledge that i…
Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel
Xin Qiu, Elliot Meyerson, Risto Miikkulainen
Neural Networks (NNs) have been extensively used for a wide spectrum of real-world regression tasks, where the goal is to predict a numerical outcome such as revenue, effectiveness…
Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains
Elliot Meyerson, Risto Miikkulainen
As deep learning applications continue to become more diverse, an interesting question arises: Can general problem solving arise from jointly learning several such diverse tasks? T…
Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimization
Santiago Gonzalez, Risto Miikkulainen
As the complexity of neural network models has grown, it has become increasingly important to optimize their design automatically through metalearning. Methods for discovering hype…
Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back
Elliot Meyerson, Risto Miikkulainen
Deep multitask learning boosts performance by sharing learned structure across related tasks. This paper adapts ideas from deep multitask learning to the setting where only a singl…
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering
Elliot Meyerson, Risto Miikkulainen
Existing deep multitask learning (MTL) approaches align layers shared between tasks in a parallel ordering. Such an organization significantly constricts the types of shared struct…