Deep reinforcement learning with spatial and temporal awareness for active boundary control of buoyancy-driven convection

Abstract

Deep reinforcement learning (DRL) applied to thermal convection control consistently produces degenerate actuation: wall-temperature policies whose outputs are saturated, pseudo-random, or spatially incoherent. Two compounding deficiencies are responsible: multilayer-perceptron policies that discard spatial flow structure, and memoryless policies that cannot distinguish self-induced flow changes from background evolution. Together they degrade the actuation into forms whose relation to the convective topology cannot be read off and which are not realisable at the actuator, even when cell coalescence (the merging of convection rolls into fewer, larger structures), which would reduce Nu, is accessible to boundary actuation. The present framework addresses both causes through four targeted design choices: convolutional policy networks, Gated Recurrent Unit (GRU) memory, off-policy training (TD3/MADDPG), and action-smoothness constraints. A systematic 2x2 factorial design isolates the contribution of each component. On Rayleigh-Benard convection at Ra = 10,000, all four configurations achieve cell coalescence and reduce Nu to as low as 1.83 (26% below the uncontrolled baseline) in 350 episodes, without the full-field data augmentation required by prior work. Crucially, coalescence is achieved even by the single-agent configuration, demonstrating that the multi-agent formulation is not a prerequisite once the policy architecture is sufficiently expressive. Applied to double-diffusive convection in the salt-finger regime, the framework spontaneously discovers a travelling-wave actuation whose phase speed adapts to the evolving mixing state of the flow, enhancing heat transfer by 19.1% and reducing salinity variance by 21.0%.