67 citations · 93 across the 21 of their papers we have counts for
23 papers
What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining
Yekun Chai, Haoyi Xiong
As pretraining increasingly repeats data, every run faces three questions: how many epochs to take, how that number should change with model size, and whether anything besides the…
Finding the Move Is Not Winning the Game: XiangqiBench for Closed-Loop Evaluation of LLM Agents
Yekun Chai, Qiwei Peng, Haoyi Xiong
Static evaluations credit a language model for naming the right move, but an agent must carry a plan through to a verified outcome while an opponent responds. We introduce XiangqiB…
ERNIE 5.0 Technical Report
Haifeng Wang, Hua Wu, Tian Wu +432
In this report, we introduce ERNIE 5.0, a natively autoregressive foundation model desinged for unified multimodal understanding and generation across text, image, video, and audio…
Understanding Subword Compositionality of Large Language Models
Qiwei Peng, Yekun Chai, Anders Søgaard
Large language models (LLMs) take sequences of subwords as input, requiring them to effective compose subword representations into meaningful word-level representations. In this pa…
Debiasing Multilingual LLMs in Cross-lingual Latent Space
Qiwei Peng, Guimin Hu, Yekun Chai +1
Debiasing techniques such as SentDebias aim to reduce bias in large language models (LLMs). Previous studies have evaluated their cross-lingual transferability by directly applying…
Curiosity-Driven Reinforcement Learning from Human Feedback
Haoran Sun, Yekun Chai, Shuohuan Wang +3
Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but often at the cost of reduced output dive…
MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
Yekun Chai, Haoran Sun, Huang Fang +3
Reinforcement learning from human feedback (RLHF) has demonstrated effectiveness in aligning large language models (LLMs) with human preferences. However, token-level RLHF suffers…
Tokenization Falling Short: On Subword Robustness in Large Language Models
Yekun Chai, Yewei Fang, Qiwei Peng +1
Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary, a process inherently sensitive to typographical errors, length varia…
Autoregressive Pre-Training on Pixels and Texts
Yekun Chai, Qingyi Liu, Jingwu Xiao +3
The integration of visual and textual information represents a promising direction in the advancement of language models. In this paper, we explore the dual modality of language--b…
On Training Data Influence of GPT Models
Yekun Chai, Qingyi Liu, Shuohuan Wang +3
Amidst the rapid advancements in generative language models, the investigation of how training data shapes the performance of GPT models is still emerging. This paper presents GPTf…
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Taishi Nakamura, Mayank Mishra, Simone Tedeschi +42
Pretrained language models are an integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim…
HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
Qiwei Peng, Yekun Chai, Xuhong Li
Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English p…
StarCoder 2 and The Stack v2: The Next Generation
Anton Lozhkov, Raymond Li, Loubna Ben Allal +63
The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership wi…
Tool-Augmented Reward Modeling
Lei Li, Yekun Chai, Shuohuan Wang +4
Reward modeling (a.k.a., preference modeling) is instrumental for aligning large language models with human preferences, particularly within the context of reinforcement learning f…
Improved Training of Mixture-of-Experts Language GANs
Yekun Chai, Qiyue Yin, Junge Zhang
Despite the dramatic success in image generation, Generative Adversarial Networks (GANs) still face great challenges in synthesizing sequences of discrete elements, in particular h…
ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models
Pengfei Zhu, Chao Pang, Yekun Chai +5
In recent years, the burgeoning interest in diffusion models has led to significant advances in image and speech generation. Nevertheless, the direct synthesis of music waveforms f…
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages
Yekun Chai, Shuohuan Wang, Chao Pang +3
Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working e…
X-PuDu at SemEval-2022 Task 6: Multilingual Learning for English and Arabic Sarcasm Detection
Yaqian Han, Yekun Chai, Shuohuan Wang +5
Detecting sarcasm and verbal irony from people's subjective statements is crucial to understanding their intended meanings and real sentiments and positions in social scenarios. Th…
Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards
Yekun Chai, Shuohuan Wang, Yu Sun +3
Derivative-free prompt learning has emerged as a lightweight alternative to prompt tuning, which only requires model inference to optimize the prompts. However, existing work did n…
RefineCap: Concept-Aware Refinement for Image Captioning
Yekun Chai, Shuo Jin, Junliang Xing
Automatically translating images to texts involves image scene understanding and language modeling. In this paper, we propose a novel model, termed RefineCap, that refines the outp…
Neural Text Classification by Jointly Learning to Cluster and Align
Yekun Chai, Haidong Zhang, Shuo Jin
Distributional text clustering delivers semantically informative representations and captures the relevance between each word and semantic clustering centroids. We extend the neura…
Highway Transformer: Self-Gating Enhanced Self-Attentive Networks
Yekun Chai, Shuo Jin, Xinwen Hou
Self-attention mechanisms have made striking state-of-the-art (SOTA) progress in various sequence learning tasks, standing on the multi-headed dot product attention by attending to…
How to Evaluate Word Representations of Informal Domain?
Yekun Chai, Naomi Saphra, Adam Lopez
Diverse word representations have surged in most state-of-the-art natural language processing (NLP) applications. Nevertheless, how to efficiently evaluate such word embeddings in…