CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision

Abstract

Recent work has shown success in incorporating pre-trained models like BERT to improve NLP systems. However, existing pre-trained models lack of causal knowledge which prevents today's NLP systems from thinking like humans. In this paper, we investigate the problem of injecting causal knowledge into pre-trained models. There are two fundamental problems: 1) how to collect various granularities of causal pairs from unstructured texts; 2) how to effectively inject causal knowledge into pre-trained models. To address these issues, we extend the idea of CausalBERT from previous studies, and conduct experiments on various datasets to evaluate its effectiveness. In addition, we adopt a regularization-based method to preserve the already learned knowledge with an extra regularization term while injecting causal knowledge. Extensive experiments on 7 datasets, including four causal pair classification tasks, two causal QA tasks and a causal inference task, demonstrate that CausalBERT captures rich causal knowledge and outperforms all pre-trained models-based state-of-the-art methods, achieving a new causal inference benchmark.

References in corpus (7)

• Overcoming catastrophic forgetting in neural networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• Guided Generation of Cause and Effect
• Temporal Common Sense Acquisition with Minimal Supervision
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting

Cited by in corpus (2)

• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• ReCo: Reliable Causal Chain Reasoning via Structural Causal Recurrent Neural Networks