CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters

Abstract

Due to the compelling improvements brought by BERT, many recent representation models adopted the Transformer architecture as their main building block, consequently inheriting the wordpiece tokenization system despite it not being intrinsically linked to the notion of Transformers. While this system is thought to achieve a good balance between the flexibility of characters and the efficiency of full words, using predefined wordpiece vocabularies from the general domain is not always suitable, especially when building models for specialized domains (e.g., the medical domain). Moreover, adopting a wordpiece tokenization shifts the focus from the word level to the subword level, making the models conceptually more complex and arguably less convenient in practice. For these reasons, we propose CharacterBERT, a new variant of BERT that drops the wordpiece system altogether and uses a Character-CNN module instead to represent entire words by consulting their characters. We show that this new model improves the performance of BERT on a variety of medical domain tasks while at the same time producing robust, word-level and open-vocabulary representations.

arXiv comment13 pages, 8 figures and 3 tables. Accepted at COLING 2020

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Training Very Deep Networks
• Character-Aware Neural Language Models
• Exploring the Limits of Language Modeling
• Enhancing Clinical Concept Extraction with Contextual Embeddings
• ERNIE: Enhanced Language Representation with Informative Entities
• Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding

Cited by in corpus (8)

• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• IIITT@LT-EDI-EACL2021-Hope Speech Detection: There is always Hope in Transformers
• Sensitive Data Detection with High-Throughput Neural Network Models for Financial Institutions
• Learning to Look Inside: Augmenting Token-Based Encoders with Character-Level Information
• Switch Point biased Self-Training: Re-purposing Pretrained Models for Code-Switching
• Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models
• Integrating Approaches to Word Representation
• Using Distributional Principles for the Semantic Study of Contextual Language Models