1 citations · 1 across the 4 of their papers we have counts for
4 papers · 1 filter
Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho +2
Data mixing decides how to combine different sources or types of data and is a consequential problem throughout language model training. In pretraining, data composition is a key d…
Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check
Nicholas Lourie, Michael Y. Hu, Kyunghyun Cho
Downstream scaling laws aim to predict task performance at larger scales from the model's performance at smaller scales. Whether such prediction should be possible is unclear: some…
Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
Michael Y. Hu, Jackson Petty, Chuan Shi +2
Pretraining language models on formal language can improve their acquisition of natural language. Which features of the formal language impart an inductive bias that leads to effec…
Safe Reinforcement Learning with Natural Language Constraints
Tsung-Yen Yang, Michael Hu, Yinlam Chow +2
While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in ma…