ShishuLM : Achieving Optimal and Efficient Parameterization with Low Attention Transformer Models

Shivanshu Kumar, Gopalakrishnan Srinivasan

While the transformer architecture has achieved state-of-the-art performance on natural language processing tasks, these models impose substantial memory and computational overhead…