CrisisBERT: a Robust Transformer for Crisis Classification and Contextual Crisis Embedding

Abstract

Classification of crisis events, such as natural disasters, terrorist attacks and pandemics, is a crucial task to create early signals and inform relevant parties for spontaneous actions to reduce overall damage. Despite crisis such as natural disasters can be predicted by professional institutions, certain events are first signaled by civilians, such as the recent COVID-19 pandemics. Social media platforms such as Twitter often exposes firsthand signals on such crises through high volume information exchange over half a billion tweets posted daily. Prior works proposed various crisis embeddings and classification using conventional Machine Learning and Neural Network models. However, none of the works perform crisis embedding and classification using state of the art attention-based deep neural networks models, such as Transformers and document-level contextual embeddings. This work proposes CrisisBERT, an end-to-end transformer-based model for two crisis classification tasks, namely crisis detection and crisis recognition, which shows promising results across accuracy and f1 scores. The proposed model also demonstrates superior robustness over benchmark, as it shows marginal performance compromise while extending from 6 to 36 events with only 51.4% additional data points. We also proposed Crisis2Vec, an attention-based, document-level contextual embedding architecture for crisis embedding, which achieve better performance than conventional crisis embedding methods such as Word2Vec and GloVe. To the best of our knowledge, our works are first to propose using transformer-based crisis classification and document-level contextual crisis embedding in the literature.

References in corpus (9)

• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• Convolutional Neural Networks for Sentence Classification
• Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Applications of Online Deep Learning for Crisis Response Using Social Media Information
• Deep Learning and Word Embeddings for Tweet Classification for Crisis Response
• Semi-supervised Discovery of Informative Tweets During the Emerging Disasters
• IPOD: An Industrial and Professional Occupations Dataset and its Applications to Occupational Data Mining and Analysis

Cited by in corpus (6)

• Network representation learning systematic review: ancestors and current development state
• Transformer-based Multi-task Learning for Disaster Tweet Categorisation
• Crisis Domain Adaptation Using Sequence-to-sequence Transformers
• EPIC30M: An Epidemics Corpus Of Over 30 Million Relevant Tweets
• Multi-task transfer learning for finding actionable information from crisis-related messages on social media
• UCD-CS at W-NUT 2020 Shared Task-3: A Text to Text Approach for COVID-19 Event Extraction on Social Media