Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science
arXiv:2110.08406 · doi:10.1038/s41467-022-31915-y
Abstract
Deep learning techniques have been increasingly applied to the natural sciences, e.g., for property prediction and optimization or material discovery. A fundamental ingredient of such approaches is the vast quantity of labelled data needed to train the model; this poses severe challenges in data-scarce settings where obtaining labels requires substantial computational or labor resources. Here, we introduce surrogate- and invariance-boosted contrastive learning (SIB-CL), a deep learning framework which incorporates three ``inexpensive'' and easily obtainable auxiliary information sources to overcome data scarcity. Specifically, these are: 1)~abundant unlabeled data, 2)~prior knowledge of symmetries or invariances and 3)~surrogate data obtained at near-zero cost. We demonstrate SIB-CL's effectiveness and generality on various scientific problems, e.g., predicting the density-of-states of 2D photonic crystals and solving the 3D time-independent Schrodinger equation. SIB-CL consistently results in orders of magnitude reduction in the number of labels needed to achieve the same network accuracies.
arXiv comment21 pages, 10 figures
References in corpus (21)
- SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap your own latent: A new approach to self-supervised Learning
- SchNet - a deep learning architecture for molecules and materials
- E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials
- Outlook for inverse design in nanophotonics
- Quantum-Chemical Insights from Deep Tensor Neural Networks
- Molecular Contrastive Learning of Representations via Graph Neural Networks
- Towards Exact Molecular Dynamics Simulations with Machine-Learned Force Fields
- What Makes for Good Views for Contrastive Learning?
- Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
- SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effects
- Inferring low-dimensional microstructure representations using convolutional neural networks
- Metallo-dielectric diamond and zinc-blende photonic crystals
- Overcoming data scarcity with transfer learning
- Density-of-States Calculations and Multiple-Scattering Theory for Photons
- Direct prediction of phonon density of states with Euclidean neural networks
- Discovering Symmetry Invariants and Conserved Quantities by Interpreting Siamese Neural Networks
- Augmentations in Hypergraph Contrastive Learning: Fabricated and Generative
- Generalized Gilat-Raubenheimer method for density-of-states calculation in photonic crystals