2 papers
cs.CL2024
Vaporetto: Efficient Japanese Tokenization Based on Improved Pointwise Linear Classification
Koichi Akabe, Shunsuke Kanda, Yusuke Oda +1
This paper proposes an approach to improve the runtime efficiency of Japanese tokenization based on the pointwise linear classification (PLC) framework, which formulates the whole…
cs.DS2024
Engineering faster double-array Aho-Corasick automata
Shunsuke Kanda, Koichi Akabe, Yusuke Oda
Multiple pattern matching in strings is a fundamental problem in text processing applications such as regular expressions or tokenization. This paper studies efficient implementati…