4 papers
Proper losses regret at least 1/2-order
Han Bao, Asuka Takatsu
A fundamental challenge in machine learning is the choice of a loss as it characterizes our learning task, is minimized in the training phase, and serves as an evaluation criterion…
Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamics
Han Bao
Contrastive learning is a self-supervised representation learning framework, where two positive views generated through data augmentation are made similar by an attraction force in…
TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
Makoto Shing, Kou Misaki, Han Bao +2
Causal language models have demonstrated remarkable capabilities, but their size poses significant challenges for deployment in resource-constrained environments. Knowledge distill…
Zipfian Whitening
Sho Yokoi, Han Bao, Hiroto Kurita +1
The word embedding space in neural models is skewed, and correcting this can improve task performance. We point out that most approaches for modeling, correcting, and measuring the…