9 citations · 12 across the 6 of their papers we have counts for
7 papers · 1 filter
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Sangwhan Moon, Daisuke Oba, Youmi Ma +2
Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investi…
Bit-level BPE: Below the byte boundary
Sangwhan Moon, Tatsuya Hiraoka, Naoaki Okazaki
Byte-level fallbacks for subword tokenization have become a common practice in large language models. In particular, it has been demonstrated to be incredibly effective as a pragma…
Two Counterexamples to Tokenization and the Noiseless Channel
Marco Cognetta, Vilém Zouhar, Sangwhan Moon +1
In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), Rényi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer w…
Learning How to Translate North Korean through South Korean
Hwichan Kim, Sangwhan Moon, Naoaki Okazaki +1
South and North Korea both use the Korean language. However, Korean NLP research has focused on South Korean only, and existing NLP systems of the Korean language, such as neural m…
StyleKQC: A Style-Variant Paraphrase Corpus for Korean Questions and Commands
Won Ik Cho, Sangwhan Moon, Jong In Kim +2
Paraphrasing is often performed with less concern for controlled style conversion. Especially for questions and commands, style-variant paraphrasing can be crucial in tone and mann…
Open Korean Corpora: A Practical Report
Won Ik Cho, Sangwhan Moon, Youngsook Song
Korean is often referred to as a low-resource language in the research community. While this claim is partially true, it is also because the availability of resources is inadequate…
Machines Getting with the Program: Understanding Intent Arguments of Non-Canonical Directives
Won Ik Cho, Young Ki Moon, Sangwhan Moon +2
Modern dialog managers face the challenge of having to fulfill human-level conversational skills as part of common user expectations, including but not limited to discourse with no…