Deceiving Google's Perspective API Built for Detecting Toxic Comments

Abstract

Social media platforms provide an environment where people can freely engage in discussions. Unfortunately, they also enable several problems, such as online harassment. Recently, Google and Jigsaw started a project called Perspective, which uses machine learning to automatically detect toxic language. A demonstration website has been also launched, which allows anyone to type a phrase in the interface and instantaneously see the toxicity score [1]. In this paper, we propose an attack on the Perspective toxic detection system based on the adversarial examples. We show that an adversary can subtly modify a highly toxic phrase in a way that the system assigns significantly lower toxicity score to it. We apply the attack on the sample phrases provided in the Perspective website and show that we can consistently reduce the toxicity scores to the level of the non-toxic phrases. The existence of such adversarial examples is very harmful for toxic detection systems and seriously undermines their usability.

4 pages

References in corpus (1)

• Adversarial Perturbations Against Deep Neural Networks for Malware Classification

Cited by in corpus (36)

• Detection of Adversarial Training Examples in Poisoning Attacks through Anomaly Detection
• Blocking Transferability of Adversarial Examples in Black-Box Learning Systems
• HotFlip: White-Box Adversarial Examples for Text Classification
• Towards Robustness Against Natural Language Word Substitutions
• The "Shut the f**k up" Phenomenon: Characterizing Incivility in Open Source Code Review Discussions
• Weight Poisoning Attacks on Pre-trained Models
• Interpretable Adversarial Perturbation in Input Embedding Space for Text
• Fooling OCR Systems with Adversarial Text Images
• Machine Learning Suites for Online Toxicity Detection
• Convolutional Neural Networks for Toxic Comment Classification
• Towards a Robust Deep Neural Network in Texts: A Survey
• Interpretable Multi-Modal Hate Speech Detection
• Evading classifiers in discrete domains with provable optimality guarantees
• A large-scale crowdsourced analysis of abuse against women journalists and politicians on Twitter
• Conversational Networks for Automatic Online Moderation
• Towards Robust Toxic Content Classification
• On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models
• Adversarial Attacks and Defense on Texts: A Survey
• PatchAttack: A Black-box Texture-based Attack with Reinforcement Learning
• Cyberbullying Detection -- Technical Report 2/2018, Department of Computer Science AGH, University of Science and Technology
• Learning to Attack: Towards Textual Adversarial Attacking in Real-world Situations
• White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks
• Robust Encodings: A Framework for Combating Adversarial Typos
• On the Social and Technical Challenges of Web Search Autosuggestion Moderation
• Efficient Combinatorial Optimization for Word-level Adversarial Textual Attack
• Adversarial Attack and Defense of Structured Prediction Models
• From Hero to Zéroe: A Benchmark of Low-Level Adversarial Attacks
• Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer
• Generating Semantically Valid Adversarial Questions for TableQA
• Characterizing Abhorrent, Misinformative, and Mistargeted Content on YouTube
• Adversarial Semantic Collisions
• Learning Neural Models for Natural Language Processing in the Face of Distributional Shift
• Visual Attack and Defense on Text
• Context-Sensitive Malicious Spelling Error Correction
• Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media
• Adversarial Machine Learning in Text Analysis and Generation