7 papers
TANGO: Token-Aggregated Nonlinear Gating Operators for Natural and Formal Language Modeling
Joshua Nunley
A standard Transformer block separates cross-token interaction in self-attention from a nonlinear feed-forward network applied independently at each position. We introduce the TANG…
Kuramoto Attention: Synchronizing Self-Attention on the Torus
Joshua Nunley
Transformer models are increasingly used as computational models of cognition and neural representation, so the mechanism implemented by self-attention is of interest beyond engine…
Attention as Frustrated Synchronization
Joshua Nunley
A network of oscillators that synchronizes perfectly computes nothing further, so an attention architecture built from synchronization must locate its computation in structured dep…
Self-Regulation through Communication in Evolved Neural Agents
Joshua Nunley
Communication is typically understood as indication: signals that transfer information from sender to receiver. We present a minimal predator avoidance task in which pairs of evolv…
Democracy on Rugged Landscapes: Phase Transitions in Optimal Voting Rules
Joshua Nunley
Laws and institutions shape individual outcomes through complex interactions with citizens' diverse circumstances, yet how different voting methods navigate this coupled landscape…
Neural Fields as World Models
Joshua Nunley
Humans rehearse possible futures offline, as in mental practice and perhaps dreaming, suggesting that world models may support task learning away from the environment. Standard mac…