2 papers
cs.LG2024
Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
Kaiyue Wen, Zhiyuan Li, Jason Wang +3
Training language models currently requires pre-determining a fixed compute budget because the typical cosine learning rate schedule depends on the total number of steps. In contra…
cs.SD2024
Anticipatory Music Transformer
John Thickstun, David Hall, Chris Donahue +1
We introduce anticipation: a method for constructing a controllable generative model of a temporal point process (the event process) conditioned asynchronously on realizations of a…