67 citations · 93 across the 21 of their papers we have counts for
7 papers · 1 filter
MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
Yekun Chai, Haoran Sun, Huang Fang +3
Reinforcement learning from human feedback (RLHF) has demonstrated effectiveness in aligning large language models (LLMs) with human preferences. However, token-level RLHF suffers…
Tokenization Falling Short: On Subword Robustness in Large Language Models
Yekun Chai, Yewei Fang, Qiwei Peng +1
Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary, a process inherently sensitive to typographical errors, length varia…
Autoregressive Pre-Training on Pixels and Texts
Yekun Chai, Qingyi Liu, Jingwu Xiao +3
The integration of visual and textual information represents a promising direction in the advancement of language models. In this paper, we explore the dual modality of language--b…
On Training Data Influence of GPT Models
Yekun Chai, Qingyi Liu, Shuohuan Wang +3
Amidst the rapid advancements in generative language models, the investigation of how training data shapes the performance of GPT models is still emerging. This paper presents GPTf…
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Taishi Nakamura, Mayank Mishra, Simone Tedeschi +42
Pretrained language models are an integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim…
HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
Qiwei Peng, Yekun Chai, Xuhong Li
Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English p…
StarCoder 2 and The Stack v2: The Next Generation
Anton Lozhkov, Raymond Li, Loubna Ben Allal +63
The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership wi…