Convolution by Evolution: Differentiable Pattern Producing Networks

Abstract

In this work we introduce a differentiable version of the Compositional Pattern Producing Network, called the DPPN. Unlike a standard CPPN, the topology of a DPPN is evolved but the weights are learned. A Lamarckian algorithm, that combines evolution and learning, produces DPPNs to reconstruct an image. Our main result is that DPPNs can be evolved/trained to compress the weights of a denoising autoencoder from 157684 to roughly 200 parameters, while achieving a reconstruction accuracy comparable to a fully connected network with more than two orders of magnitude more parameters. The regularization ability of the DPPN allows it to rediscover (approximate) convolutional network architectures embedded within a fully connected architecture. Such convolutional architectures are the current state of the art for many computer vision applications, so it is satisfying that DPPNs are capable of discovering this structure rather than having to build it in by design. DPPNs exhibit better generalization when tested on the Omniglot dataset after being trained on MNIST, than directly encoded fully connected autoencoders. DPPNs are therefore a new framework for integrating learning and evolution.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Predicting Parameters in Deep Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Neural GPUs Learn Algorithms
• Generative NeuroEvolution for Deep Learning
• Phenotypic Plasticity, the Baldwin Effect, and the Speeding up of Evolution: the Computational Roots of an Illusion

Cited by in corpus (11)

• AutoML: A Survey of the State-of-the-Art
• Large-Scale Evolution of Image Classifiers
• A Particle Swarm Optimization-based Flexible Convolutional Auto-Encoder for Image Classification
• Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks
• A Genetic Programming Approach to Designing Convolutional Neural Network Architectures
• Meta-learning autoencoders for few-shot prediction
• A Flexible Approach to Automated RNN Architecture Generation
• Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
• Breaking the Activation Function Bottleneck through Adaptive Parameterization
• Epigenetic evolution of deep convolutional models
• Evolving Differentiable Gene Regulatory Networks