Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines

Murtaza Rangwala, Richard O. Sinnott, Rajkumar Buyya

Between them, the members of a research group or a circle of friends own several consumer computers, none large enough to run a capable large language model. Existing systems pool…