Splitwiser: Efficient LM inference with constrained resources

Asad Aali, Adney Cardoza, Melissa Capo

Efficient inference of LLMs remains a crucial challenge, with two main phases: a compute-intensive prompt computation and a memory-intensive token generation. Despite existing batc…