MorphPiece : A Linguistic Tokenizer for Large Language Models

Haris Jabbar

Tokenization is a critical part of modern NLP pipelines. However, contemporary tokenizers for Large Language Models are based on statistical analysis of text corpora, without much…