SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference

Harish KB, Jagadeeswaran M, Pradheep P +2

Running large language models (LLMs) locally continues to be limited by restrictions of compute and memory on consumer hardware. The popular acceleration technologies, such as quan…