2 papers
cs.CL2024
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich +10
We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlie…
cs.AR2024
PipeOrgan: Efficient Inter-operation Pipelining with Flexible Spatial Organization and Interconnects
Raveesh Garg, Hyoukjun Kwon, Eric Qin +3
Because of the recent trends in Deep Neural Networks (DNN) models being memory-bound, inter-operator pipelining for DNN accelerators is emerging as a promising optimization. Inter-…