From Roofline to Ruggedness: Decomposing and Smoothing the GEMM Performance Landscape

Abstract

Adjacent GEMM problems that differ by a single 128-element step in N can show 30% different throughput. This pervasive performance ruggedness - invisible to roofline analysis and peak-FLOPs intuition, yet dominant for every non-peak workload - is the subject of this paper. We propose performance ruggedness analysis, an analytical framework complementary to roofline: rather than summarizing a GPU with a scalar bound, it treats the full multidimensional performance surface as the object of study, decomposes its texture into mechanism-attributable components, and separates software-removable from hardware-bound losses. The framing is analogous to deep-learning loss landscapes: a continuous quantity (idealized time 2MNK/peak) made rugged by discrete hardware substrates (tiles, sub-groups, cache lines, DRAM channels). We instantiate it on BF16 NN GEMM on Intel Battlemage (Arc B580, sycl-tla) via a 32,768-configuration sweep over (M,N,K) in {128,...,4096}^3. We introduce roughness, the mean absolute step-to-step throughput change, which starts at 16.8 TFLOPs/128-step against an ideal of 2.0. A two-stage stack - best-of-six dynamic tile selection and a novel dynamic-programming padding-and-splitting optimizer (precomputed once, O(1) at runtime) - cuts roughness by 70% and raises mean throughput by 30%. Cross-tile experiments show the residual sawtooth period scales exactly with the tile size, ruling out cache conflicts and attributing the rest to four hardware-bound sources. Finally, we derive the optimal achievable landscape from first principles - datasheet integers alone, no kernel run or simulator - and turn it into an optimality scale (Kernel Optimality Levels) grading any kernel by how much of that landscape it attains and how close its roughness lies to the hardware floor; the production kernel and our optimized stack rate L0 and L2 despite both reporting ~95% of peak.