Doc2Dict: Information Extraction as Text Generation

Abstract

Typically, information extraction (IE) requires a pipeline approach: first, a sequence labeling model is trained on manually annotated documents to extract relevant spans; then, when a new document arrives, a model predicts spans which are then post-processed and standardized to convert the information into a database entry. We replace this labor-intensive workflow with a transformer language model trained on existing database records to directly generate structured JSON. Our solution removes the workload associated with producing token-level annotations and takes advantage of a data source which is generally quite plentiful (e.g. database records). As long documents are common in information extraction tasks, we use gradient checkpointing and chunked encoding to apply our method to sequences of up to 32,000 tokens on a single GPU. Our Doc2Dict approach is competitive with more complex, hand-engineered pipelines and offers a simple but effective baseline for document-level information extraction. We release our Doc2Dict model and code to reproduce our experiments and facilitate future work.

References in corpus (9)

• Sequence to Sequence Learning with Neural Networks
• Natural Language Processing (almost) from Scratch
• Linformer: Self-Attention with Linear Complexity
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Training Deep Nets with Sublinear Memory Cost
• The Natural Language Decathlon: Multitask Learning as Question Answering
• A Simple Language Model for Task-Oriented Dialogue
• Rapid Adaptation of BERT for Information Extraction on Domain-Specific Business Documents
• SeqGenSQL -- A Robust Sequence Generation Model for Structured Query Language