Inference Auctions

Keegan Harris, Siddharth Prasad, Asher Trockman +2

When inference demand exceeds available compute capacity, model providers must decide which requests should be served first. Users have different tolerances for delay from an LLM A…