4 papers · 1 filter
PMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Tasks
Srijith Nair, Atilla Eryilmaz, Jia Liu
In the literature, many continual learning (CL) algorithms have been proposed to address the issue of catastrophic forgetting in ML models (i.e., learning new tasks leads to the lo…
PowLU: An Activation Function for Stable Pre-Training of LLMs
Peijie Jiang, Yuqi Feng, Cunyin Peng +5
In contemporary large language models (LLMs), the swish-gated linear unit (SwiGLU) activation function is widely adopted to regulate the information flow and introduce non-linearit…
SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling
Quanen Sun, Changxin Tian, Ke Shi +6
Scaling laws guide large language model training by relating compute to cross-entropy loss, and recent work further extends them to predict downstream benchmark performance. Howeve…
Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale
Liujie Zhang, Benzhe Ning, Rui Yang +8
Reinforcement learning (RL) post-training has proven effective at unlocking reasoning, self-reflection, and tool-use capabilities in large language models. As models extend to omni…