Molecule Attention Transformer

Abstract

Designing a single neural network architecture that performs competitively across a range of molecule property prediction tasks remains largely an open challenge, and its solution may unlock a widespread use of deep learning in the drug discovery industry. To move towards this goal, we propose Molecule Attention Transformer (MAT). Our key innovation is to augment the attention mechanism in Transformer using inter-atomic distances and the molecular graph structure. Experiments show that MAT performs competitively on a diverse set of molecular prediction tasks. Most importantly, with a simple self-supervised pretraining, MAT requires tuning of only a few hyperparameter values to achieve state-of-the-art performance on downstream tasks. Finally, we show that attention weights learned by MAT are interpretable from the chemical point of view.

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Neural Message Passing for Quantum Chemistry
• Molecular Graph Convolutions: Moving Beyond Fingerprints
• Quantum-Chemical Insights from Deep Tensor Neural Networks
• AtomNet: A Deep Convolutional Neural Network for Bioactivity Prediction in Structure-based Drug Discovery
• Neural Natural Language Inference Models Enhanced with External Knowledge
• Chemception: A Deep Neural Network with Minimal Chemistry Knowledge Matches the Performance of Expert-developed QSAR/QSPR Models
• Do Attention Heads in BERT Track Syntactic Dependencies?
• Learning Graph-Level Representation for Drug Discovery
• Three-Dimensionally Embedded Graph Convolutional Network (3DGCN) for Molecule Interpretation
• Edge Attention-based Multi-Relational Graph Convolutional Networks
• Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Models
• Self-Attention Based Molecule Representation for Predicting Drug-Target Interaction
• Learning to SMILE(S)
• A survey on Big Data and Machine Learning for Chemistry

Cited by in corpus (17)

• Molecular Contrastive Learning of Representations via Graph Neural Networks
• ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction
• A Comprehensive Survey on Deep Graph Representation Learning
• Do Transformers Really Perform Bad for Graph Representation?
• Molecular representation learning with language models and domain-relevant auxiliary tasks
• Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning
• KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction
• Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction
• Dual-view Molecule Pre-training
• Heterogeneous Graph Neural Networks with Post-hoc Explanations for Multi-modal and Explainable Land Use Inference
• Learning Attributed Graph Representations with Communicative Message Passing Transformer
• Graph-Aware Transformer: Is Attention All Graphs Need?
• Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction
• Structure-aware Interactive Graph Neural Networks for the Prediction of Protein-Ligand Binding Affinity
• PACIA: Parameter-Efficient Adapter for Few-Shot Molecular Property Prediction
• GeoT: A Geometry-aware Transformer for Reliable Molecular Property Prediction and Chemically Interpretable Representation Learning
• Predicting times of waiting on red signals using BERT