5 papers
Content-Based Addressing for Long Context
Mahesh Godavarti
Rotary position embedding (RoPE) uses each token's integer position to determine the rotation applied inside attention. This works well for local token order, but increasing contex…
Labeled Incidence Structures for Native Transformer Modeling of Text, Knowledge Graphs, and Hypergraphs
Mahesh Godavarti
Current Transformer interfaces index tokens by one or more integer coordinates, which determine their addresses inside attention. In RoPE and its multi-axis or hierarchical variant…
Journey Operators for Structured Multi-Axis Composition
Mahesh Godavarti
Many kinds of data have structure along one or more axes: words in a sentence, pixels in an image, nodes in a tree, frames in audio, or cells in a 3D volume. Along one axis, order…
Cumsum-Composable Phase Transport for Low-Cost Streaming Keyword Spotting
Mahesh Godavarti
State-space sequence models are attractive for streaming speech because they maintain compact recurrent state, but scan-style training kernels can have unfavorable constants for sh…
Diversity vs Degrees of Freedom in Gaussian Fading Channels
Mahesh Godavarti
The classical definitions extract degrees of freedom (DOF) via $C(\snr)/\log\snr$ and diversity (DIV) via $-\log P_e(\snr)/\log\snr$, using $\log\snr$ as the common gauge for both.…