5 papers
DIFFLET: A Fast Inference Engine for Diffusion Transformer Models on AWS Trainium
Bin Ma, Min Yu, Vijay Kodamalla +4
On AWS Trainium2, every component of a diffusion model runs as a program compiled ahead of time for fixed shapes, while pipelines differ across models, prompt lengths across reques…
A Multi-Engine Dataflow for MoE Decoding on AWS Tranium
Bin Ma, Wenjie Fan, Jialin Liu +2
Mixture-of-Experts (MoE) decoding on scratchpad-based tensor accelerators (STA) is dominated by moving expert weights while the compute engines sit idle. This traffic is hard to hi…
Bayesian Repetition Penalty: A Principled Adjacent-Conditional Framework for Reversing Attention Collapse in Autoregressive Language Models
Wenjie Fan, Bin Ma, Dong Li
Attention collapse in autoregressive language models -- manifested as repetitive token loops where the model becomes trapped in self-reinforcing attractors -- is a persistent patho…
Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization
Jiu Chen, Shuangyan Yang, Xu Xiong +4
Decentralized LLM inference distributes computation among heterogeneous nodes across the internet, offering a performant and cost-efficient solution, alternative to traditional cen…
Machine Learning-Guided Memory Optimization for DLRM Inference on Tiered Memory
Jie Ren, Bin Ma, Shuangyan Yang +4
Deep learning recommendation models (DLRMs) are widely used in industry, and their memory capacity requirements reach the terabyte scale. Tiered memory architectures provide a cost…