Neo: A Learned Query Optimizer

Abstract

Query optimization is one of the most challenging problems in database systems. Despite the progress made over the past decades, query optimizers remain extremely complex components that require a great deal of hand-tuning for specific workloads and datasets. Motivated by this shortcoming and inspired by recent advances in applying machine learning to data management challenges, we introduce Neo (Neural Optimizer), a novel learning-based query optimizer that relies on deep neural networks to generate query executions plans. Neo bootstraps its query optimization model from existing optimizers and continues to learn from incoming queries, building upon its successes and learning from its failures. Furthermore, Neo naturally adapts to underlying data patterns and is robust to estimation errors. Experimental results demonstrate that Neo, even when bootstrapped from a simple optimizer like PostgreSQL, can learn a model that offers similar performance to state-of-the-art commercial optimizers, and in some cases even surpass them.

References in corpus (16)

• Efficient Estimation of Word Representations in Vector Space
• Deep Learning in Neural Networks: An Overview
• Toward Multimodal Image-to-Image Translation
• Deep Q-learning from Demonstrations
• Convolutional Neural Networks over Tree Structures for Programming Language Processing
• Deep Reinforcement Learning for Join Order Enumeration
• Thinking Fast and Slow with Deep Learning and Tree Search
• Learned Cardinalities: Estimating Correlated Joins with Deep Learning
• Database Meets Deep Learning: Challenges and Opportunities
• HoloDetect: Few-Shot Learning for Error Detection
• Innateness, AlphaZero, and Artificial Intelligence
• Learning to Optimize Join Queries With Deep Reinforcement Learning
• Towards a Hands-Free Query Optimizer through Deep Learning
• Learning Scheduling Algorithms for Data Processing Clusters
• LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations
• Cuttlefish: A Lightweight Primitive for Adaptive Query Processing

Cited by in corpus (58)

• ALEX: An Updatable Adaptive Learned Index
• Bao: Learning to Steer Query Optimizers
• Deep Unsupervised Cardinality Estimation
• Are We Ready For Learned Cardinality Estimation?
• Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction
• Balsa: Learning a Query Optimizer Without Expert Demonstrations
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Are Updatable Learned Indexes Ready?
• QuickSel: Quick Selectivity Learning with Mixture Models
• Black or White? How to Develop an AutoTuner for Memory-based Analytics [Extended Version]
• Towards Dynamic and Safe Configuration Tuning for Cloud Databases
• Learning Scheduling Algorithms for Data Processing Clusters
• RadixSpline: A Single-Pass Learned Index
• NeuroCard: One Cardinality Estimator for All Tables
• AdaChain: A Learned Adaptive Blockchain
• NeurDB: An AI-powered Autonomous Data System
• Deploying a Steered Query Optimizer in Production at Microsoft
• Monotonic Cardinality Estimation of Similarity Selection: A Deep Learning Approach
• DeepDB: Learn from Data, not from Queries!
• A View on Deep Reinforcement Learning in System Optimization
• FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation
• MLOS: An Infrastructure for Automated Software Performance Engineering
• Efficient Query Re-optimization with Judicious Subquery Selections
• Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing
• Consistent and Flexible Selectivity Estimation for High-Dimensional Data
• Flow-Loss: Learning Cardinality Estimates That Matter
• Toward Code Generation: A Survey and Lessons from Semantic Parsing
• Hype or Heuristic? Quantum Reinforcement Learning for Join Order Optimisation
• Optimizing Dataflow Systems for Scalable Interactive Visualization
• Phoebe: A Learning-based Checkpoint Optimizer
• Blueprinting the Cloud: Unifying and Automatically Optimizing Cloud Data Infrastructures with BRAD -- Extended Version
• A Unified Transferable Model for ML-Enhanced DBMS
• LMKG: Learned Models for Cardinality Estimation in Knowledge Graphs
• FOSS: A Self-Learned Doctor for Query Optimizer
• Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings
• Updatable Learned Index with Precise Positions
• Progressive Neural Index Search for Database System
• One Model to Rule them All: Towards Zero-Shot Learning for Databases
• A Pluggable Learned Index Method via Sampling and Gap Insertion
• Join Query Optimization with Deep Reinforcement Learning Algorithms
• Low Rank Learning for Offline Query Optimization
• ML-AQP: Query-Driven Approximate Query Processing based on Machine Learning
• Efficient Deep Learning Pipelines for Accurate Cost Estimations Over Large Scale Query Workload
• A Novel Technique for Query Plan Representation Based on Graph Neural Nets
• Are Joins over LSM-Trees Ready? Take RocksDB as an Example
• Maliva: Using Machine Learning to Rewrite Visualization Queries Under Time Constraints
• Boosting Cloud Data Analytics using Multi-Objective Optimization
• Is Your Learned Query Optimizer Behaving As You Expect? A Machine Learning Perspective
• A Comparative Exploration of ML Techniques for Tuning Query Degree of Parallelism
• Mosaic: A Sample-Based Database System for Open World Query Processing
• Online Sketch-based Query Optimization
• Can Deep Neural Networks Predict Data Correlations from Column Names?
• Uncertainty-aware Cardinality Estimation by Neural Network Gaussian Process
• Simpli-Squared: A Very Simple Yet Unexpectedly Powerful Join Ordering Algorithm Without Cardinality Estimates
• Experience-Enhanced Learning: One Size Still does not Fit All in Automatic Database
• Efficient RDF Graph Storage based on Reinforcement Learning
• Wield: Systematic Reinforcement Learning With Progressive Randomization
• Multiple Query Optimization using a Hybrid Approach of Classical and Quantum Computing