5 papers
Uncertainty Awareness on Unsupervised Domain Adaptation for Time Series Data
Weide Liu, Xiaoyang Zhong, Lu Wang +4
Unsupervised domain adaptation methods seek to generalize effectively on unlabeled test data, especially when encountering the common challenge in time series data that distributio…
Learning Dynamics in Continual Pre-Training for Large Language Models
Xingjin Wang, Howe Tissue, Lu Wang +2
Continual Pre-Training (CPT) has become a popular and effective method to apply strong foundation models to specific downstream tasks. In this work, we explore the learning dynamic…
Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training
Mozhi Zhang, Howe Tissue, Lu Wang +1
We introduce~\textsc{Domain2Vec}, a novel approach that decomposes any dataset into a linear combination of several \emph{meta-domains}, a new concept designed to capture the key u…
Scaling Law with Learning Rate Annealing
Howe Tissue, Venus Wang, Lu Wang
We find that the cross-entropy loss curves of neural language models empirically adhere to a scaling law with learning rate (LR) annealing over training steps: $$L(s) = L_0 + A\cdo…
A Mallows-like Criterion for Anomaly Detection with Random Forest Implementation
Gaoxiang Zhao, Lu Wang, Xiaoqiang Wang
The effectiveness of anomaly signal detection can be significantly undermined by the inherent uncertainty of relying on one specified model. Under the framework of model average me…