4 papers · 1 filter
Deep Kernel Fusion for Transformers
Zixi Zhang, Zhiwen Mo, Yiren Zhao +1
Agentic LLM inference with long contexts is increasingly limited by memory bandwidth rather than compute. In this setting, SwiGLU MLP blocks, whose large weights exceed cache capac…
Hardware and Software Platform Inference
Cheng Zhang, Hanna Foerster, Robert D. Mullins +2
It is now a common business practice to buy access to large language model (LLM) inference rather than self-host, because of significant upfront hardware infrastructure and energy…
Optimised Grouped-Query Attention Mechanism for Transformers
Yuang Chen, Cheng Zhang, Xitong Gao +3
Grouped-query attention (GQA) has been widely adopted in LLMs to mitigate the complexity of multi-head attention (MHA). To transform an MHA to a GQA, neighbour queries in MHA are e…
Unlocking the Global Synergies in Low-Rank Adapters
Zixi Zhang, Cheng Zhang, Xitong Gao +3
Low-rank Adaption (LoRA) has been the de-facto parameter-efficient fine-tuning technique for large language models. We present HeteroLoRA, a light-weight search algorithm that leve…