1 paper
Benzi Busigin, Yuval Pinter
Transformer language models process input provided as subword fragments, but natural language semantics usually rely on word-level concepts. Detokenization is the process where mod…