8 citations · 8 across the 2 of their papers we have counts for
3 papers
cs.CL2026
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Sangwhan Moon, Daisuke Oba, Youmi Ma +2
Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investi…
cs.CL2026★ 8 cited
Open Korean Corpora: A Practical Report
Won Ik Cho, Sangwhan Moon, Youngsook Song
Korean is often referred to as a low-resource language in the research community. While this claim is partially true, it is also because the availability of resources is inadequate…
cs.CL2025
Bit-level BPE: Below the byte boundary
Sangwhan Moon, Tatsuya Hiraoka, Naoaki Okazaki
Byte-level fallbacks for subword tokenization have become a common practice in large language models. In particular, it has been demonstrated to be incredibly effective as a pragma…