2 papers
cs.LG2026
DoSA: Dots Sparse Attention
Fanxu Meng, Taosong Fang
Sparse attention visits few keys, yet finding them can still require scoring every key with every indexer head. We introduce DoSA (Dots Sparse Attention), a lossless method for mak…
cs.CL2024
BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching
Zhen Zheng, Xin Ji, Taosong Fang +3
Large language models (LLMs) increasingly play an important role in a wide range of information processing and management tasks in industry. Many of these tasks are performed in la…