Unsolved Problems in ML Safety

Abstract

Machine learning (ML) systems are rapidly increasing in size, are acquiring new capabilities, and are increasingly deployed in high-stakes settings. As with other powerful technologies, safety for ML should be a leading research priority. In response to emerging safety challenges in ML, such as those introduced by recent large-scale models, we provide a new roadmap for ML Safety and refine the technical problems that the field needs to address. We present four problems ready for research, namely withstanding hazards ("Robustness"), identifying hazards ("Monitoring"), reducing inherent model hazards ("Alignment"), and reducing systemic hazards ("Systemic Safety"). Throughout, we clarify each problem's motivation and provide concrete research directions.

arXiv commentPosition Paper

References in corpus (60)

• Deep Learning with Differential Privacy
• Intriguing properties of neural networks
• Learning Transferable Visual Models From Natural Language Supervision
• Language Models are Few-Shot Learners
• On the Opportunities and Risks of Foundation Models
• Equality of Opportunity in Supervised Learning
• On Calibration of Modern Neural Networks
• Evaluating Large Language Models Trained on Code
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Theoretically Principled Trade-off between Robustness and Accuracy
• Certified Defenses against Adversarial Examples
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Deep Learning Scaling is Predictable, Empirically
• Deep Anomaly Detection with Outlier Exposure
• Cooperative Inverse Reinforcement Learning
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Tent: Fully Test-time Adaptation by Entropy Minimization
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Quantifying Generalization in Reinforcement Learning
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
• Motivating the Rules of the Game for Adversarial Example Research
• Scaling Laws for Autoregressive Generative Modeling
• Scalable agent alignment via reward modeling: a research direction
• RobustBench: a standardized adversarial robustness benchmark
• Blind Backdoors in Deep Learning Models
• Fixing Data Augmentation to Improve Adversarial Robustness
• Smooth Adversarial Training
• Testing Robustness Against Unforeseen Adversaries
• SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection
• You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion
• A 20-Year Community Roadmap for Artificial Intelligence Research in the US
• Unsupervised Translation of Programming Languages
• Characterizing Adversarial Examples Based on Spatial Consistency Information for Semantic Segmentation
• Detecting and Characterizing Lateral Phishing at Scale
• Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation
• Alignment of Language Agents
• Measuring and Improving Consistency in Pretrained Language Models
• AI safety via debate
• Program Synthesis with Large Language Models
• AI Research Considerations for Human Existential Safety (ARCHES)
• Risks from Learned Optimization in Advanced Machine Learning Systems
• What are you optimizing for? Aligning Recommender Systems with Human Values
• The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models
• Handcrafted Backdoors in Deep Neural Networks
• Poisoning and Backdooring Contrastive Learning
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• Avoiding Side Effects By Considering Future Tasks
• Open Problems in Cooperative AI
• Certifiably Adversarially Robust Detection of Out-of-Distribution Data
• Reinforcement Learning Under Moral Uncertainty
• SafeLife 1.0: Exploring Side Effects in Complex Environments
• Conservative Objective Models for Effective Offline Model-Based Optimization
• Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks
• Hidden Incentives for Auto-Induced Distributional Shift
• What Would Jiminy Cricket Do? Towards Agents That Behave Morally
• Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Results
• Augmenting Decision Making via Interactive What-If Analysis

Cited by in corpus (11)

• On the Opportunities and Risks of Foundation Models
• Unleashing the potential of prompt engineering for large language models
• Worldwide AI Ethics: a review of 200 guidelines and recommendations for AI governance
• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• Deception Abilities Emerged in Large Language Models
• AI Safety for Everyone
• A General Language Assistant as a Laboratory for Alignment
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• Overparameterized Linear Regression under Adversarial Attacks
• Cycle Consistency-based Uncertainty Quantification of Neural Networks in Inverse Imaging Problems
• Certified Adversarial Defenses Meet Out-of-Distribution Corruptions: Benchmarking Robustness and Simple Baselines