activity
20232026
most citedFlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

16 citations · 17 across the 4 of their papers we have counts for

collaborators
Showing 2023Show all

1 paper · 1 filter

cs.LG2023★ 1 cited

A Case Study in CUDA Kernel Fusion: Implementing FlashAttention-2 on NVIDIA Hopper Architecture using the CUTLASS Library

Ganesh Bikshandi, Jay Shah

We provide an optimized implementation of the forward pass of FlashAttention-2, a popular memory-aware scaled dot-product attention algorithm, as a custom fused CUDA kernel targeti…