Generating Synthetic Text Data to Evaluate Causal Inference Methods
arXiv:2102.05638
Abstract
Drawing causal conclusions from observational data requires making assumptions about the true data-generating process. Causal inference research typically considers low-dimensional data, such as categorical or numerical fields in structured medical records. High-dimensional and unstructured data such as natural language complicates the evaluation of causal inference methods; such evaluations rely on synthetic datasets with known causal effects. Models for natural language generation have been widely studied and perform well empirically. However, existing methods not immediately applicable to producing synthetic datasets for causal evaluations, as they do not allow for quantifying a causal effect on the text itself. In this work, we develop a framework for adapting existing generation models to produce synthetic text datasets with known causal effects. We use this framework to perform an empirical comparison of four recently-proposed methods for estimating causal effects from text data. We release our code and synthetic datasets.
References in corpus (15)
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Matching Methods for Causal Inference: A Review and a Look Forward
- Supervised Topic Models
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Generating Multi-label Discrete Patient Records using Generative Adversarial Networks
- Adapting Text Embeddings for Causal Inference
- How to Make Causal Inferences Using Texts
- Unifying Human and Statistical Evaluation for Natural Language Generation
- On Measurement Bias in Causal Inference
- Benchmarking Framework for Performance-Evaluation of Causal Inference Analysis
- The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data
- Atlantic Causal Inference Conference (ACIC) Data Analysis Challenge 2017
- Towards the Creation of a Large Corpus of Synthetically-Identified Clinical Notes
- RealCause: Realistic Causal Inference Benchmarking