3 papers
cs.LG2024
ResidualDroppath: Enhancing Feature Reuse over Residual Connections
Sejik Park
Residual connections are one of the most important components in neural network architectures for mitigating the vanishing gradient problem and facilitating the training of much de…
cs.LG2024
Learning More Generalized Experts by Merging Experts in Mixture-of-Experts
Sejik Park
We observe that incorporating a shared layer in a mixture-of-experts can lead to performance degradation. This leads us to hypothesize that learning shared features poses challenge…
cs.AI2024
Diverse Feature Learning by Self-distillation and Reset
Sejik Park
Our paper addresses the problem of models struggling to learn diverse features, due to either forgetting previously learned features or failing to learn new ones. To overcome this…