The Extremum Stack as Optimal Memory for Rate-Independent Sequence Models: Information-Theoretic Foundations and Online Complexity
arXiv:2606.05245
The paper shows that the extremum stack used in Preisach attention fully characterizes rate‑independent sequence models and is the minimal representation both in Kolmogorov and Shannon senses, while also analyzing its online update costs and providing efficient implementations.
Abstract
Rate-independent sequence models respond to the ordered structure of input extrema rather than to absolute timing or token position. This principle underlies Preisach Attention, a hysteretic alternative to softmax attention in which sequence history is represented by a stack of alternating extrema generated by the classical wiping-out rule. This paper establishes the information-theoretic and online algorithmic foundations of that representation. We prove that the Preisach extremum stack is not merely a convenient implementation detail, but the complete invariant of computable rate-independent sequence functionals: a functional is rate-independent if and only if it factors through the stack. We then show that the stack is minimal in two complementary senses. In the Kolmogorov setting, the shortest exact representation answering all rate-independent queries has complexity equal to that of the stack up to an additive constant independent of sequence length and stack depth. In the Shannon setting, under any input distribution, every sufficient representation contains at least as much mutual information about the input as the stack, with equality only for representations informationally equivalent to it. Finally, we analyse the online maintenance cost of this minimal state. Although the standard stack update is amortised constant time, adversarial inputs can induce linear worst-case latency. We prove a matching output-change lower bound, show that binary search reduces boundary detection but not deletion, and give an exact finger-tree implementation with worst-case (O(\log k)) update time, where (k) is the current stack depth. The results provide a principled foundation for replacing full sequence histories or KV-cache-like memories by extremum-stack states in rate-independent neural architectures, without approximation and with bounded online latency.
31 pages, 1 figure. This manuscript consolidates and extends the results previously developed in arXiv:2605.18885, arXiv:2605.23603, and arXiv:2606.05245 into a unified study of Preisach extremum stacks