A generative pre-trained transformer with Kerr-soliton attention

Abstract

Generative pre-trained transformers (GPTs) have enabled capability-rich language models, but their operation incurs substantial costs in computation, memory, and data movement. A major source of this cost is attention, a nonlinear transformation that computes context-dependent weights from token correlations. Here, we introduce Kerr-soliton attention, harnessing driven-dissipative nonlinear dynamics in a resonator to realize, execute, and validate deep-learning attention in physical hardware. We train a GPT language model using an analytic representation of Kerr-soliton attention and explore generative inference by streaming model-produced inputs through a time-multiplexed soliton ensemble, whose nonlinear evolution transforms the scores into attention weights. We observe high-fidelity agreement between the experimentally produced weights and those predicted by the model. By mapping memory and nonlinear computation to physical dynamics, Kerr-soliton attention points toward hybrid digital-physical systems that reduce data movement in deep learning.