6 papers
ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inference
Han Meng, Danny Willow Liu, Dong Li
Layerwise offloading reduces the GPU memory footprint of large diffusion transformer (DiT) inference by prefetching upcoming layers from host memory, but its effectiveness hinges o…
HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer
Qi Cai, Jingwen Chen, Chengmin Gao +22
The evolution of visual generative models has long been constrained by fragmented architectures relying on disjoint text encoders and external VAEs. In this report, we present HiDr…
PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding
Zihao An, Taichi Liu, Ziqiong Liu +3
Speculative decoding accelerates Large Language Models (LLMs) inference by using a lightweight draft model to propose candidate tokens that are verified in parallel by the target m…
NeuronMLP: Efficient LLM Inference via Singular Value Decomposition Compression and Tiling on AWS Trainium
Dinghong Song, Jierui Xu, Weichu Yang +2
Emerging AI accelerators have started to gain attention and offer new opportunities for efficient inference of large language models (LLMs). Trainium, an AI accelerator recently de…
Adversarial Contrastive Learning for LLM Quantization Attacks
Dinghong Song, Zhiwei Xu, Hai Wan +3
Model quantization is critical for deploying large language models (LLMs) on resource-constrained hardware, yet recent work has revealed severe security risks that benign LLMs in f…
AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache
Dinghong Song, Yuan Feng, Yiwei Wang +6
Large Language Models (LLMs) are widely used in generative applications such as chatting, code generation, and reasoning. However, many realworld workloads such as classification,…