7 papers
Attending to Memory Writes Reduces Seed Variability in Recurrent Length Generalization
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
Recurrent memory requires learning useful updates, but attention added to assist learning can also compromise length generalization. We study bounded delta attention: causal attent…
SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
Attention provides direct access to past representations, but retaining an ever-growing history is costly. Recurrent models bound persistent state, yet must preserve selected infor…
Consolidator: Learning Persistent Routed Memory Across Context Boundaries
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
Copying short-term memory (STM) into a slower store can preserve state across a context boundary, but persistence alone does not ensure that the retained state influences subsequen…
Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\…
Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
Modern deep neural networks rely on Euclidean scalar activations (e.g., ReLU) and global normalization techniques (e.g., LayerNorm) to prevent gradient instability in deep architec…
Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
For over a decade, explicit memory architectures like the Neural Turing Machine have remained theoretically appealing yet practically intractable for language modeling due to catas…
CaliciBoost: Performance-Driven Evaluation of Molecular Representations for Caco-2 Permeability Prediction
Huong Van Le, Weibin Ren, Junhong Kim +8
Caco-2 permeability serves as a critical in vitro indicator for predicting the oral absorption of drug candidates during early-stage drug discovery. To enhance the accuracy and eff…