4 citations · 6 across the 5 of their papers we have counts for
Showing 2024Show all
2 papers · 1 filter
cs.CL2024
Tokenization Falling Short: On Subword Robustness in Large Language Models
Yekun Chai, Yewei Fang, Qiwei Peng +1
Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary, a process inherently sensitive to typographical errors, length varia…
cs.CL2024★ 4 cited
HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
Qiwei Peng, Yekun Chai, Xuhong Li
Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English p…