On Robustness of Neural Semantic Parsers
arXiv:2102.01563
Abstract
Semantic parsing maps natural language (NL) utterances into logical forms (LFs), which underpins many advanced NLP problems. Semantic parsers gain performance boosts with deep neural networks, but inherit vulnerabilities against adversarial examples. In this paper, we provide the empirical study on the robustness of semantic parsers in the presence of adversarial attacks. Formally, adversaries of semantic parsing are considered to be the perturbed utterance-LF pairs, whose utterances have exactly the same meanings as the original ones. A scalable methodology is proposed to construct robustness test sets based on existing benchmark corpora. Our results answered five research questions in measuring the sate-of-the-art parsers' performance on robustness test sets, and evaluating the effect of data augmentation.
arXiv commentLong Paper, Accepted to EACL2021
References in corpus (13)
- Robustness May Be at Odds with Accuracy
- Adversarial Training Methods for Semi-Supervised Text Classification
- On Adversarial Examples for Character-Level Neural Machine Translation
- Language to Logical Form with Neural Attention
- On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- Coarse-to-Fine Decoding for Neural Semantic Parsing
- Sequence-to-Action: End-to-End Semantic Graph Generation for Semantic Parsing
- A Survey on Semantic Parsing
- On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models
- Metamorphic Relation Based Adversarial Attacks on Differentiable Neural Computer
- Elephant in the Room: An Evaluation Framework for Assessing Adversarial Examples in NLP
- Question Generation from SQL Queries Improves Neural Semantic Parsing