4 papers · 1 filter
Consolidator: Learning Persistent Routed Memory Across Context Boundaries
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
Copying short-term memory (STM) into a slower store can preserve state across a context boundary, but persistence alone does not ensure that the retained state influences subsequen…
Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\…
Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
Modern deep neural networks rely on Euclidean scalar activations (e.g., ReLU) and global normalization techniques (e.g., LayerNorm) to prevent gradient instability in deep architec…
Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
For over a decade, explicit memory architectures like the Neural Turing Machine have remained theoretically appealing yet practically intractable for language modeling due to catas…