4 papers
Lossless Vocabulary Reduction for Auto-Regressive Language Models
Daiki Chijiwa, Taku Hasegawa, Kyosuke Nishida +4
Tokenization -- the process of decomposing a given text into a sequence of subwords called tokens -- is one of the key components in the development of language models. Particularl…
The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms
Hikari Otsuka, Daiki Chijiwa, Yasuyuki Okoshi +3
The strong lottery ticket hypothesis (SLTH) conjectures that high-performing subnetworks, called strong lottery tickets (SLTs), are hidden in randomly initialized neural networks.…
Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models
Daiki Chijiwa, Taku Hasegawa, Kyosuke Nishida +2
While foundation models have been exploited for various expert tasks through fine-tuning, any foundation model will become outdated due to its old knowledge or limited capability.…
Partially Frozen Random Networks Contain Compact Strong Lottery Tickets
Hikari Otsuka, Daiki Chijiwa, Ãngel López GarcÃa-Arias +6
Randomly initialized dense networks contain subnetworks that achieve high accuracy without weight learning--strong lottery tickets (SLTs). Recently, Gadhikar et al. (2023) demonstr…