263 citations · 730 across the 19 of their papers we have counts for
8 papers · 1 filter
Large Language Models as Optimizers
Chengrun Yang, Xuezhi Wang, Yifeng Lu +4
Optimization is ubiquitous. While derivative-based algorithms have been powerful tools for various problems, the absence of gradient imposes challenges on many real-world applicati…
Simple synthetic data reduces sycophancy in large language models
Jerry Wei, Da Huang, Yifeng Lu +2
Sycophancy is an undesirable behavior where models tailor their responses to follow a human user's view even when that view is not objectively correct (e.g., adapting liberal views…
Brainformers: Trading Simplicity for Efficiency
Yanqi Zhou, Nan Du, Yanping Huang +12
Transformers are central to recent successes in natural language processing and computer vision. Transformers have a mostly uniform backbone where layers alternate between feed-for…
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
Sang Michael Xie, Hieu Pham, Xuanyi Dong +7
The mixture proportions of pretraining data domains (e.g., Wikipedia, books, web text) greatly affect language model (LM) performance. In this paper, we propose Domain Reweighting…
Symbol tuning improves in-context learning in language models
Jerry Wei, Le Hou, Andrew Lampinen +8
We present symbol tuning - finetuning language models on in-context input-label pairs where natural language labels (e.g., "positive/negative sentiment") are replaced with arbitrar…
Larger language models do in-context learning differently
Jerry Wei, Jason Wei, Yi Tay +8
We study how in-context learning (ICL) in language models is affected by semantic priors versus input-label mappings. We investigate two setups-ICL with flipped labels and ICL with…