106 citations · 342 across the 14 of their papers we have counts for
16 papers
Contextual Experience Replay for Self-Improvement of Language Agents
Yitao Liu, Chenglei Si, Karthik Narasimhan +1
Large language model (LLM) agents have been applied to sequential decision-making tasks such as web navigation, but without any environment-specific experiences, they often fail in…
Predicting Empirical AI Research Outcomes with Language Models
Jiaxin Wen, Chenglei Si, Yueh-han Chen +2
Many promising-looking ideas in AI research fail to deliver, but their validation takes substantial human labor and compute. Predicting an idea's chance of success is thus crucial…
Configurable Foundation Models: Building LLMs from a Modular Perspective
Chaojun Xiao, Zhengyan Zhang, Chenyang Song +20
Advancements in LLMs have recently unveiled challenges tied to computational efficiency and continual scalability due to their requirements of huge parameters, making the applicati…
Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations
Chenglei Si, Dan Friedman, Nitish Joshi +3
In-context learning (ICL) is an important paradigm for adapting large language models (LLMs) to new tasks, but the generalization behavior of ICL remains poorly understood. We inve…
Getting MoRE out of Mixture of Language Model Reasoning Experts
Chenglei Si, Weijia Shi, Chen Zhao +2
While recent large language models (LLMs) improve on various question answering (QA) datasets, it remains difficult for a single model to generalize across question types that requ…
READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises
Chenglei Si, Zhengyan Zhang, Yingfa Chen +3
For many real-world applications, the user-generated inputs usually contain various noises due to speech recognition errors caused by linguistic variations1 or typographical errors…
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop, :, Teven Le Scao +391
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to wi…
Prompting GPT-3 To Be Reliable
Chenglei Si, Zhe Gan, Zhengyuan Yang +4
Large language models (LLMs) show impressive abilities via few-shot prompting. Commercialized APIs such as OpenAI GPT-3 further increase their use in real-world language applicatio…
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky +8
What are the units of text that we want to model? From bytes to multi-word expressions, text can be analyzed and generated at many granularities. Until recently, most natural langu…
What's in a Name? Answer Equivalence For Open-Domain Question Answering
Chenglei Si, Chen Zhao, Jordan Boyd-Graber
A flaw in QA evaluation is that annotations often only provide one gold answer. Thus, model predictions semantically equivalent to the answer but superficially different are consid…
Adversarial Training for Machine Reading Comprehension with Virtual Embeddings
Ziqing Yang, Yiming Cui, Chenglei Si +4
Adversarial training (AT) as a regularization method has proved its effectiveness on various tasks. Though there are successful applications of AT on some NLP tasks, the distinguis…
Sub-Character Tokenization for Chinese Pretrained Language Models
Chenglei Si, Zhengyan Zhang, Yingfa Chen +6
Tokenization is fundamental to pretrained language models (PLMs). Existing tokenization methods for Chinese PLMs typically treat each character as an indivisible token. However, th…
Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning
Chenglei Si, Zhengyan Zhang, Fanchao Qi +4
Pretrained language models (PLMs) perform poorly under adversarial attacks. To improve the adversarial robustness, adversarial data augmentation (ADA) has been widely adopted to co…
CharBERT: Character-aware Pre-trained Language Model
Wentao Ma, Yiming Cui, Chenglei Si +3
Most pre-trained language models (PLMs) construct word representations at subword level with Byte-Pair Encoding (BPE) or its variations, by which OOV (out-of-vocab) words are almos…
Benchmarking Robustness of Machine Reading Comprehension Models
Chenglei Si, Ziqing Yang, Yiming Cui +3
Machine Reading Comprehension (MRC) is an important testbed for evaluating models' natural language understanding (NLU) ability. There has been rapid progress in this area, with ne…
What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
Chenglei Si, Shuohang Wang, Min-Yen Kan +1
Multiple-Choice Reading Comprehension (MCRC) requires the model to read the passage and question, and select the correct answer among the given options. Recent state-of-the-art mod…