A Survey of Deep Learning for Scientific Discovery

Abstract

Over the past few years, we have seen fundamental breakthroughs in core problems in machine learning, largely driven by advances in deep neural networks. At the same time, the amount of data collected in a wide array of scientific domains is dramatically increasing in both size and complexity. Taken together, this suggests many exciting opportunities for deep learning applications in scientific settings. But a significant challenge to this is simply knowing where to start. The sheer breadth and diversity of different deep learning techniques makes it difficult to determine what scientific problems might be most amenable to these methods, or which specific combination of methods might offer the most promising first approach. In this survey, we focus on addressing this central issue, providing an overview of many widely used deep learning models, spanning visual, sequential and graph structured data, associated tasks and different training methods, along with techniques to use deep learning with less data and better interpret these complex models --- two central considerations for many scientific use cases. We also include overviews of the full design process, implementation tips, and links to a plethora of tutorials, research summaries and open-sourced deep learning pipelines and pretrained models, developed by the community. We hope that this survey will help accelerate the use of deep learning across different scientific domains.

References in corpus (90)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Continuous control with deep reinforcement learning
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• YOLOv3: An Incremental Improvement
• mixup: Beyond Empirical Risk Minimization
• Neural Message Passing for Quantum Chemistry
• WaveNet: A Generative Model for Raw Audio
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Axiomatic Attribution for Deep Networks
• Teaching Machines to Read and Comprehend
• Improved Regularization of Convolutional Neural Networks with Cutout
• Unsupervised Domain Adaptation by Backpropagation
• Learning Important Features Through Propagating Activation Differences
• Conditional Image Generation with PixelCNN Decoders
• Predicting Cardiovascular Risk Factors from Retinal Fundus Photographs using Deep Learning
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Attention-Based Models for Speech Recognition
• Molecular Graph Convolutions: Moving Beyond Fingerprints
• Learning deep representations by mutual information estimation and maximization
• Pixel Recurrent Neural Networks
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Semi-Supervised Learning with Deep Generative Models
• Temporal Ensembling for Semi-Supervised Learning
• A Neural Conversational Model
• Understanding Neural Networks Through Deep Visualization
• NICE: Non-linear Independent Components Estimation
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• A Deep Reinforced Model for Abstractive Summarization
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Snorkel: Rapid Training Data Creation with Weak Supervision
• SmoothGrad: removing noise by adding noise
• Density estimation using Real NVP
• Object Detectors Emerge in Deep Scene CNNs
• Learning Representations by Maximizing Mutual Information Across Views
• Efficient Neural Architecture Search via Parameter Sharing
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Interaction Networks for Learning about Objects, Relations and Physics
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Group Equivariant Convolutional Networks
• Detecting Cancer Metastases on Gigapixel Pathology Images
• Deep learning tools for the measurement of animal behavior in neuroscience
• Data Augmentation using Random Image Cropping and Patching for Deep CNNs
• Regularizing and Optimizing LSTM Language Models
• MolGAN: An implicit generative model for small molecular graphs
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• A DIRT-T Approach to Unsupervised Domain Adaptation
• Similarity of Neural Network Representations Revisited
• Massively Multitask Networks for Drug Discovery
• Do ImageNet Classifiers Generalize to ImageNet?
• Large Scale Adversarial Representation Learning
• DeepNAT: Deep Convolutional Neural Network for Segmenting Neuroanatomy
• Some Improvements on Deep Convolutional Neural Network Based Image Classification
• InterpretML: A Unified Framework for Machine Learning Interpretability
• What makes ImageNet good for transfer learning?
• Learning how to explain neural networks: PatternNet and PatternAttribution
• Self-training with Noisy Student improves ImageNet classification
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Unsupervised Domain Adaptation through Self-Supervision
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Parameter-Efficient Transfer Learning for NLP
• Convergent Learning: Do different neural networks learn the same representations?
• An Analysis of Neural Language Modeling at Multiple Scales
• Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach
• Domain Adaptive Transfer Learning with Specialist Models
• Learning Conditional Deformable Templates with Convolutional Networks
• Dive into Deep Learning
• Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
• Revisiting Self-Supervised Visual Representation Learning
• A Two-Step Graph Convolutional Decoder for Molecule Generation
• VisualBackProp: efficient visualization of CNNs
• Off-Policy Policy Gradient with State Distribution Correction
• End-to-End Learning on 3D Protein Structure for Interface Prediction
• Structured Training for Neural Network Transition-Based Parsing
• Identifying and Controlling Important Neurons in Neural Machine Translation
• Data Recombination for Neural Semantic Parsing
• Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation
• Semi-Supervised Sequence Modeling with Cross-View Training
• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
• Deep Recurrent Neural Network for Protein Function Prediction from Sequence
• Image Reconstruction with Predictive Filter Flow
• Investigating Multilingual NMT Representations at Scale
• A deep generative model for gene expression profiles from single-cell RNA sequencing

Cited by in corpus (19)

• Medical Deep Learning -- A systematic Meta-Review
• Benchmarking Graph Neural Networks
• The data-driven future of high energy density physics
• Review: Deep Learning in Electron Microscopy
• Learning the Travelling Salesperson Problem Requires Rethinking Generalization
• Time Series Analysis and Modeling to Forecast: a Survey
• Single-exposure absorption imaging of ultracold atoms using deep learning
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
• Explicit and implicit measures of emotions: Data-science might help to account for data complexity and heterogeneity
• Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads
• Using Machine Learning to Emulate Agent-Based Simulations
• Livewired Neural Networks: Making Neurons That Fire Together Wire Together
• Accenture at CheckThat! 2021: Interesting claim identification and ranking with contextually sensitive lexical training data augmentation
• Deep Learning -- A first Meta-Survey of selected Reviews across Scientific Disciplines, their Commonalities, Challenges and Research Impact
• Itsy Bitsy SpiderNet: Fully Connected Residual Network for Fraud Detection
• Leveraging Uncertainty from Deep Learning for Trustworthy Materials Discovery Workflows
• Incremental Cross-Domain Adaptation for Robust Retinopathy Screening via Bayesian Deep Learning
• Ordinal Regression with Fenton-Wilkinson Order Statistics: A Case Study of an Orienteering Race
• Toward Building Science Discovery Machines