5 papers
Beyond Accuracy: Prefix-Invariant Realizations of Low-Precision Fast Matrix Multiplication
Shuxiao Xie, Shuyang Xie, Yuan Cao +3
Fast matrix multiplication saves multiplications through exact cancellation, but rounding sums that mix token rows can leave contributions from later tokens in earlier language mod…
Backward-State Policy Is Part of the Learning Algorithm
Shuxiao Xie, Shuyang Xie, Dezhi Ran +2
Low-precision training rounds tensors that the backward pass reads again, often for several gradients; each use can read the forward's rounded value, the original, or a new random…
Fast Matrix Multiplication in fp8: Certified Coefficient Optimization and Measured Error
Shuxiao Xie, Shuyang Xie, Yuan Cao +3
A Strassen-type algorithm has many realizations with the same exact product and multiplication count yet different fp8 error because basis changes reshape coefficient geometry, pos…
One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse
Shuxiao Xie, Shuyang Xie, Yuan Cao +3
A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalization disrupts their compounding. D…
KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Bandits
Dezhi Ran, Shuxiao Xie, Mingfang Ji +9
High-performance GPU kernels are critical for efficient LLM serving, yet their optimization remains a bottleneck requiring deep system expertise. While code LLMs show promise in ge…