From the 1 of 8 linked papers with an AI index.
8 papers
AMUSE: Anytime Muon with Stable Gradient Evaluation
Jueun Kim, Baekrok Shin, Jihun Yun +3
The paper introduces AMUSE, an optimization method that combines Muon's orthogonal momentum updates with Schedule-Free iterate averaging to achieve fast, stable training without le…
Label-Efficient Dataset Pruning via Semi-Supervised Pseudo-Labeling
Yeseul Cho, Baekrok Shin, Changmin Kang +1
Dataset pruning reduces the storage and training costs of deep learning by selecting an informative subset from a large dataset. However, most existing pruning methods require full…
Nesterov Acceleration with Operator Decomposition
Jaewook Lee, Ernest K. Ryu, Chulhee Yun
We propose Nesterov acceleration with Operator Decomposition (NOD), which extends Nesterov's accelerated gradient descent (NAG) from smooth strongly convex optimization to the broa…
Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime
Beomhan Baek, Minhak Song, Chulhee Yun
Adam [Kingma & Ba, 2015] is the de facto optimizer in deep learning, yet its theoretical understanding remains limited. Prior analyses show that Adam favors solutions aligned with…
Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training
Minhak Song, Beomhan Baek, Kwangjun Ahn +1
As both model and dataset sizes continue to scale rapidly, conventional pretraining strategies with fixed compute budgets-such as cosine learning rate schedules-are increasingly in…
Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More
Geonhui Yoo, Minhak Song, Chulhee Yun
When training deep neural networks with gradient descent, sharpness often increases -- a phenomenon known as progressive sharpening -- before saturating at the edge of stability. A…