InhibiDistilbert: Knowledge Distillation for a ReLU and Addition-based Transformer

Tony Zhang, Rickard Brännvall

This work explores optimizing transformer-based language models by integrating model compression techniques with inhibitor attention, a novel alternative attention mechanism. Inhib…