Generating Semantically Valid Adversarial Questions for TableQA
arXiv:2005.12696
Abstract
Adversarial attack on question answering systems over tabular data (TableQA) can help evaluate to what extent they can understand natural language questions and reason with tables. However, generating natural language adversarial questions is difficult, because even a single character swap could lead to huge semantic difference in human perception. In this paper, we propose SAGE (Semantically valid Adversarial GEnerator), a Wasserstein sequence-to-sequence model for TableQA white-box attack. To preserve meaning of original questions, we apply minimum risk training with SIMILE and entity delexicalization. We use Gumbel-Softmax to incorporate adversarial loss for end-to-end training. Our experiments show that SAGE outperforms existing local attack models on semantic validity and fluency while achieving a good attack success rate. Finally, we demonstrate that adversarial training with SAGE augmented data can improve performance and robustness of TableQA systems.
arXiv commentAAAI 2021 Workshop on Towards Robust, Secure and Efficient Machine Learning
References in corpus (20)
- Auto-Encoding Variational Bayes
- Categorical Reparameterization with Gumbel-Softmax
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
- Effective Approaches to Attention-based Neural Machine Translation
- Generating Natural Adversarial Examples
- Deep Text Classification Can be Fooled
- Adversarial Examples for Evaluating Reading Comprehension Systems
- Deceiving Google's Perspective API Built for Detecting Toxic Comments
- A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization
- Compositional Semantic Parsing on Semi-Structured Tables
- Adversarial NLI: A New Benchmark for Natural Language Understanding
- X-SQL: reinforce schema representation with context
- Robust Neural Machine Translation with Doubly Adversarial Inputs
- Combating Adversarial Misspellings with Robust Word Recognition
- Adversarial TableQA: Attention Supervision for Question Answering on Tables
- On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models
- Generating Natural Language Adversarial Examples on a Large Scale with Generative Models
- Detecting Overfitting via Adversarial Examples
- Beyond BLEU: Training Neural Machine Translation with Semantic Similarity