Hopscotch: Discovering and Skipping Redundancies in Language Models

Mustafa Eyceoz, Nikhil Shivakumar Nayak, Hao Wang +2

Modern causal language models stack many attention blocks to improve performance, but not all blocks are necessary for every task. We propose Hopscotch, a simple yet effective meth…