Deriving Neural Architectures from Sequence and Graph Kernels

Abstract

The design of neural architectures for structured objects is typically guided by experimental insights rather than a formal process. In this work, we appeal to kernels over combinatorial structures, such as sequences and graphs, to derive appropriate neural operations. We introduce a class of deep recurrent neural operations and formally characterize their associated kernel spaces. Our recurrent modules compare the input to virtual reference objects (cf. filters in CNN) via the kernels. Similar to traditional neural operations, these reference objects are parameterized and directly optimized in end-to-end training. We empirically evaluate the proposed class of neural architectures on standard applications such as language modeling and molecular graph regression, achieving state-of-the-art results across these applications.

extended version of ICML 2017 camera ready

References in corpus (17)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• LSTM: A Search Space Odyssey
• Improving neural networks by preventing co-adaptation of feature detectors
• Neural Architecture Search with Reinforcement Learning
• Spectral Networks and Locally Connected Networks on Graphs
• Recurrent Neural Network Regularization
• Deep Convolutional Networks on Graph-Structured Data
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory
• Discriminative Embeddings of Latent Variable Models for Structured Data
• A Convolutional Neural Network for Modelling Sentences
• Pointer Sentinel Mixture Models
• Gated Graph Sequence Neural Networks
• Long Short-Term Memory-Networks for Machine Reading
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Recurrent Additive Networks
• Deep Convolutional Networks are Hierarchical Kernel Machines

Cited by in corpus (8)

• A Comprehensive Survey on Deep Graph Representation Learning
• Syntax-Directed Variational Autoencoder for Structured Data
• Adversarial Attack on Graph Structured Data
• Learning Multimodal Graph-to-Graph Translation for Molecular Optimization
• Graph Capsule Convolutional Neural Networks
• Graph Kernels: A Survey
• Stability and Generalization of Graph Convolutional Neural Networks
• Learning Universal Graph Neural Network Embeddings With Aid Of Transfer Learning