Communication Compression for Tensor Parallel LLM Inference

Jan Hansen-Palmus, Michael Truong Le, Oliver Hausdörfer +1

Large Language Models (LLMs) have pushed the frontier of artificial intelligence but are comprised of hundreds of billions of parameters and operations. For faster inference latenc…