ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning

Abstract

Data is a critical asset in AI, as high-quality datasets can significantly improve the performance of machine learning models. In safety-critical domains such as autonomous vehicles, offline deep reinforcement learning (offline DRL) is frequently used to train models on pre-collected datasets, as opposed to training these models by interacting with the real-world environment as the online DRL. To support the development of these models, many institutions make datasets publicly available with opensource licenses, but these datasets are at risk of potential misuse or infringement. Injecting watermarks to the dataset may protect the intellectual property of the data, but it cannot handle datasets that have already been published and is infeasible to be altered afterward. Other existing solutions, such as dataset inference and membership inference, do not work well in the offline DRL scenario due to the diverse model behavior characteristics and offline setting constraints. In this paper, we advocate a new paradigm by leveraging the fact that cumulative rewards can act as a unique identifier that distinguishes DRL models trained on a specific dataset. To this end, we propose ORL-AUDITOR, which is the first trajectory-level dataset auditing mechanism for offline RL scenarios. Our experiments on multiple offline DRL models and tasks reveal the efficacy of ORL-AUDITOR, with auditing accuracy over 95% and false positive rates less than 2.88%. We also provide valuable insights into the practical implementation of ORL-AUDITOR by studying various parameter settings. Furthermore, we demonstrate the auditing capability of ORL-AUDITOR on open-source datasets from Google and DeepMind, highlighting its effectiveness in auditing published datasets. ORL-AUDITOR is open-sourced at https://github.com/link-zju/ORL-Auditor.

To appear in the Network and Distributed System Security Symposium (NDSS) 2024, San Diego, CA, USA

References in corpus (18)

• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Offline Reinforcement Learning with Implicit Q-Learning
• Robust Deep Reinforcement Learning with Adversarial Attacks
• Deep Reinforcement Learning for Robotic Manipulation-The state of the art
• Hyperparameter Selection for Offline Reinforcement Learning
• STARDATA: A StarCraft AI Research Dataset
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Dataset Inference: Ownership Resolution in Machine Learning
• Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture
• Locally Private Distributed Reinforcement Learning
• Open-sourced Dataset Protection via Backdoor Watermarking
• Benchmarking Offline Reinforcement Learning on Real-Robot Hardware
• ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning
• PrivTrace: Differentially Private Trajectory Synthesis by Adaptive Markov Model
• On the Privacy Risks of Cell-Based NAS Architectures
• FACE-AUDITOR: Data Auditing in Facial Recognition Systems
• MIAShield: Defending Membership Inference Attacks via Preemptive Exclusion of Members
• AHEAD: Adaptive Hierarchical Decomposition for Range Query under Local Differential Privacy

Cited by in corpus (1)

• ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning