Showing 2024Show all
3 papers · 1 filter
cs.CL2024
Zipfian Whitening
Sho Yokoi, Han Bao, Hiroto Kurita +1
The word embedding space in neural models is skewed, and correcting this can improve task performance. We point out that most approaches for modeling, correcting, and measuring the…
stat.ML2024
Proper losses regret at least 1/2-order
Han Bao, Asuka Takatsu
A fundamental challenge in machine learning is the choice of a loss as it characterizes our learning task, is minimized in the training phase, and serves as an evaluation criterion…
stat.ML2024
Self-attention Networks Localize When QK-eigenspectrum Concentrates
Han Bao, Ryuichiro Hataya, Ryo Karakida
The self-attention mechanism prevails in modern machine learning. It has an interesting functionality of adaptively selecting tokens from an input sequence by modulating the degree…