Learning Equivariant Representations

Abstract

State-of-the-art deep learning systems often require large amounts of data and computation. For this reason, leveraging known or unknown structure of the data is paramount. Convolutional neural networks (CNNs) are successful examples of this principle, their defining characteristic being the shift-equivariance. By sliding a filter over the input, when the input shifts, the response shifts by the same amount, exploiting the structure of natural images where semantic content is independent of absolute pixel positions. This property is essential to the success of CNNs in audio, image and video recognition tasks. In this thesis, we extend equivariance to other kinds of transformations, such as rotation and scaling. We propose equivariant models for different transformations defined by groups of symmetries. The main contributions are (i) polar transformer networks, achieving equivariance to the group of similarities on the plane, (ii) equivariant multi-view networks, achieving equivariance to the group of symmetries of the icosahedron, (iii) spherical CNNs, achieving equivariance to the continuous 3D rotation group, (iv) cross-domain image embeddings, achieving equivariance to 3D rotations for 2D inputs, and (v) spin-weighted spherical CNNs, generalizing the spherical CNNs and achieving equivariance to 3D rotations for spherical vector fields. Applications include image classification, 3D shape classification and retrieval, panoramic image classification and segmentation, shape alignment and pose estimation. What these models have in common is that they leverage symmetries in the data to reduce sample and model complexity and improve generalization performance. The advantages are more significant on (but not limited to) challenging tasks where data is limited or input perturbations such as arbitrary rotations are present.

References in corpus (39)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Semi-Supervised Classification with Graph Convolutional Networks
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• HEALPix -- a Framework for High Resolution Discretization, and Fast Analysis of Data Distributed on the Sphere
• Geometric deep learning: going beyond Euclidean data
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Spectral Networks and Locally Connected Networks on Graphs
• Multi-Scale Context Aggregation by Dilated Convolutions
• cuDNN: Efficient Primitives for Deep Learning
• Rotation-invariant convolutional neural networks for galaxy morphology prediction
• Deep Convolutional Inverse Graphics Network
• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• Group Equivariant Convolutional Networks
• Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Deformable Convolutional Networks
• Making Convolutional Networks Shift-Invariant Again
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• Rotation equivariant vector field networks
• Exploiting Cyclic Symmetry in Convolutional Neural Networks
• Bag of Tricks for Image Classification with Convolutional Neural Networks
• Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
• Angular velocity of gravitational radiation from precessing binaries and the corotating frame
• SO-Net: Self-Organizing Network for Point Cloud Analysis
• Spherical CNNs on Unstructured Grids
• Explorations in Homeomorphic Variational Auto-Encoding
• 3D-Aware Scene Manipulation via Inverse Graphics
• Fast and Exact Spin-s Spherical Harmonic Transforms
• SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation
• How should spin-weighted spherical functions be defined?
• Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision
• Theoretical Aspects of Group Equivariant Neural Networks
• Meta-Learning Symmetries by Reparameterization
• Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View
• Cross-Domain 3D Equivariant Image Embeddings
• Fast, exact (but unstable) spin spherical harmonic transforms
• Labeling Panoramas with Spherical Hourglass Networks