Infusing clinical knowledge into tokenisers for language models

Abul Hasan, Jinge Wu, Quang Ngoc Nguyen +7

This study introduces a novel knowledge enhanced tokenisation mechanism, K-Tokeniser, for clinical text processing. Technically, at initialisation stage, K-Tokeniser populates glob…