Meta-Learning Symmetries by Reparameterization

Abstract

Many successful deep learning architectures are equivariant to certain transformations in order to conserve parameters and improve generalization: most famously, convolution layers are equivariant to shifts of the input. This approach only works when practitioners know the symmetries of the task and can manually construct an architecture with the corresponding equivariances. Our goal is an approach for learning equivariances from data, without needing to design custom task-specific architectures. We present a method for learning and encoding equivariances into networks by learning corresponding parameter sharing patterns from data. Our method can provably represent equivariance-inducing parameter sharing for any finite group of symmetry transformations. Our experiments suggest that it can automatically learn to encode equivariances to common transformations used in image processing tasks. We provide our experiment code at https://github.com/AllanYangZhou/metalearning-symmetries.

ICLR 2021

References in corpus (28)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• Theoretical Models of Learning to Learn
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• AutoAugment: Learning Augmentation Policies from Data
• Group Equivariant Convolutional Networks
• Hamiltonian Neural Networks
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• Meta-Learning with Latent Embedding Optimization
• Gauge Equivariant Convolutional Networks and the Icosahedral CNN
• Equivariance Through Parameter-Sharing
• General $E(2)$-Equivariant Steerable CNNs
• Generalized Inner Loop Meta-Learning
• How to train your MAML
• Lagrangian Neural Networks
• Deep Scale-spaces: Equivariance Over Scale
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Learning Invariances using the Marginal Likelihood
• Modular meta-learning
• Torchmeta: A Meta-Learning library for PyTorch
• Learning Invariances in Neural Networks
• Revisiting Spatial Invariance with Low-Rank Local Connectivity

Cited by in corpus (13)

• Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future
• SymmetryGAN: Symmetry Discovery with Deep Learning
• Automatic Symmetry Discovery with Lie Algebra Convolutional Network
• Towards Learning Convolutions from Scratch
• Modularity in Deep Learning: A Survey
• Embedding Adaptation is Still Needed for Few-Shot Learning
• Addressing the Topological Defects of Disentanglement via Distributed Operators
• Equivariant Learning of Stochastic Fields: Gaussian Processes and Steerable Conditional Neural Processes
• Grounding inductive biases in natural images:invariance stems from variations in data
• A Foliated View of Transfer Learning
• Learning Augmentation Distributions using Transformed Risk Minimization
• Identifying Physical Law of Hamiltonian Systems via Meta-Learning
• Learning Equivariant Representations