7 papers
TANGO: Token-Aggregated Nonlinear Gating Operators for Natural and Formal Language Modeling
Joshua Nunley
A standard Transformer block separates cross-token interaction in self-attention from a nonlinear feed-forward network applied independently at each position. We introduce the TANG…
Kuramoto Attention: Synchronizing Self-Attention on the Torus
Joshua Nunley
Transformer models are increasingly used as computational models of cognition and neural representation, so the mechanism implemented by self-attention is of interest beyond engine…
Attention as Frustrated Synchronization
Joshua Nunley
A network of oscillators that synchronizes perfectly computes nothing further, so an attention architecture built from synchronization must locate its computation in structured dep…
Self-Regulation through Communication in Evolved Neural Agents
Joshua Nunley
Communication is typically understood as indication: signals that transfer information from sender to receiver. We present a minimal predator avoidance task in which pairs of evolv…
Democracy on Rugged Landscapes: Phase Transitions in Optimal Voting Rules
Joshua Nunley
Laws and institutions shape individual outcomes through complex interactions with citizens' diverse circumstances, yet how different voting methods navigate this coupled landscape…
Subgroups of Induce Natural RNN and Transformer Architectures
Joshua Nunley
This paper presents a direct framework for sequence models with hidden states on closed subgroups of U(d). We use a minimal axiomatic setup and derive recurrent and transformer tem…