17 citations · 27 across the 3 of their papers we have counts for
3 papers · 1 filter
Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
Vasu Shyam, Anna Golubeva, Quentin Anthony
We present tensor and sequence parallelism (TSP), a parallel execution strategy that folds tensor parallelism and sequence parallelism onto a single device axis. In conventional mu…
Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
Bo Peng, Daniel Goldstein, Quentin Anthony +27
We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) architecture. Our architectural design advancements include multi-headed matrix-value…
RWKV: Reinventing RNNs for the Transformer Era
Bo Peng, Eric Alcaide, Quentin Anthony +31
Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence leng…