Validating UTF-8 In Less Than One Instruction Per Byte

Abstract

The majority of text is stored in UTF-8, which must be validated on ingestion. We present the lookup algorithm, which outperforms UTF-8 validation routines used in many libraries and languages by more than 10 times using commonly available SIMD instructions. To ensure reproducibility, our work is freely available as open source software.

References in corpus (4)

• The ARM Scalable Vector Extension
• Parsing Gigabytes of JSON per Second
• Faster Base64 Encoding and Decoding Using AVX2 Instructions
• Base64 encoding and decoding at almost the speed of a memory copy

Cited by in corpus (6)

• On-Demand JSON: A Better Way to Parse Documents?
• Parsing Millions of URLs per Second
• Unicode at Gigabytes per Second
• Transcoding Unicode Characters with AVX-512 Instructions
• Scanning HTML at Tens of Gigabytes per Second on ARM Processors
• Transcoding Billions of Unicode Characters per Second with SIMD Instructions