4 papers
Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections
Elian Iluk, Gil Ben-Artzi
Mixture-of-Experts (MoE) scales language models by routing each input through a small set of independently parameterized experts. We show that copying this design into convolutiona…
Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation
Uri Z. Kialy, Gil Ben-Artzi
Parameter-Efficient Fine-Tuning (PEFT) has become the de facto standard for adapting Vision Transformers (ViTs) to downstream tasks. While parameter count has been the dominant eff…
LayerDropBack: A Universally Applicable Approach for Accelerating Training of Deep Networks
Evgeny Hershkovitch Neiterman, Gil Ben-Artzi
Training very deep convolutional networks is challenging, requiring significant computational resources and time. Existing acceleration methods often depend on specific architectur…
ChannelDropBack: Forward-Consistent Stochastic Regularization for Deep Networks
Evgeny Hershkovitch Neiterman, Gil Ben-Artzi
Incorporating stochasticity into the training process of deep convolutional networks is a widely used technique to reduce overfitting and improve regularization. Existing technique…