6 papers
PMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Tasks
Srijith Nair, Atilla Eryilmaz, Jia Liu
In the literature, many continual learning (CL) algorithms have been proposed to address the issue of catastrophic forgetting in ML models (i.e., learning new tasks leads to the lo…
SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling
Quanen Sun, Changxin Tian, Ke Shi +5
Scaling laws guide large language model training by relating compute to cross-entropy loss, and recent work further extends them to predict downstream benchmark performance. Howeve…
PowLU: An Activation Function for Stable Pre-Training of LLMs
Peijie Jiang, Yuqi Feng, Cunyin Peng +5
In contemporary large language models (LLMs), the swish-gated linear unit (SwiGLU) activation function is widely adopted to regulate the information flow and introduce non-linearit…
Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale
Liujie Zhang, Benzhe Ning, Rui Yang +8
Reinforcement learning (RL) post-training has proven effective at unlocking reasoning, self-reflection, and tool-use capabilities in large language models. As models extend to omni…
FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation
Srijith Nair, Michael Lin, Peizhong Ju +3
Collaborative training methods like Federated Learning (FL) and Split Learning (SL) enable distributed machine learning without sharing raw data. However, FL assumes clients can tr…
PSMGD: Periodic Stochastic Multi-Gradient Descent for Fast Multi-Objective Optimization
Mingjing Xu, Peizhong Ju, Jia Liu +1
Multi-objective optimization (MOO) lies at the core of many machine learning (ML) applications that involve multiple, potentially conflicting objectives (e.g., multi-task learning,…