An Information Extraction Study: Take In Mind the Tokenization!

Abstract

Current research on the advantages and trade-offs of using characters, instead of tokenized text, as input for deep learning models, has evolved substantially. New token-free models remove the traditional tokenization step; however, their efficiency remains unclear. Moreover, the effect of tokenization is relatively unexplored in sequence tagging tasks. To this end, we investigate the impact of tokenization when extracting information from documents and present a comparative study and analysis of subword-based and character-based models. Specifically, we study Information Extraction (IE) from biomedical texts. The main outcome is twofold: tokenization patterns can introduce inductive bias that results in state-of-the-art performance, and the character-based models produce promising results; thus, transitioning to token-free IE models is feasible.

arXiv commentSubmitted Manuscript/Preprint (accepted at EUSFLAT 2023, to be published in Lecture Notes in Computer Science (LNCS))

References in corpus (5)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Joint entity recognition and relation extraction as a multi-head selection problem
• Span-based Joint Entity and Relation Extraction with Transformer Pre-training
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning

Cited by in corpus (3)

• Deep Learning based Key Information Extraction from Business Documents: Systematic Literature Review
• Enhancing Biomedical Knowledge Discovery for Diseases: An Open-Source Framework Applied on Rett Syndrome and Alzheimer's Disease
• Evaluating the Predictive Features of Person-Centric Knowledge Graph Embeddings: Unfolding Ablation Studies