Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation

Buu Phan, Ashish Khisti, Karen Ullrich

Computing next-token likelihood ratios between two language models (LMs) is a standard task in training paradigms such as knowledge distillation. Since this requires both models to…