EXECUTE: A Multilingual Benchmark for LLM Token Understanding

Lukas Edman, Helmut Schmid, Alexander Fraser

The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE.…