Mapping Natural Language Commands to Web Elements

Abstract

The web provides a rich, open-domain environment with textual, structural, and spatial properties. We propose a new task for grounding language in this environment: given a natural language command (e.g., "click on the second article"), choose the correct element on the web page (e.g., a hyperlink or text box). We collected a dataset of over 50,000 commands that capture various phenomena such as functional references (e.g. "find who made this site"), relational reasoning (e.g. "article by john"), and visual reasoning (e.g. "top-most article"). We also implemented and analyzed three baseline models that capture different phenomena present in the dataset.

arXiv commentEMNLP 2018

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Dynamic Coattention Networks For Question Answering
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• Show and Tell: A Neural Image Caption Generator
• Compositional Semantic Parsing on Semi-Structured Tables
• From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood
• Reasoning About Pragmatics with Neural Listeners and Speakers
• Alignment-based compositional semantics for instruction following