Robust Recurrent Reinforcement Learning under Evolving Hidden Disturbances with Application to Rover Wheel Slip

Abstract

Reinforcement learning (RL) performs well in continuous-control tasks, but evolving hidden disturbances create partial observability: the agent must infer decision-relevant latent dynamics from interaction history. This study investigates how observation history, action history, history length, and network structure affect recurrent Twin Delayed Deep Deterministic Policy Gradient (TD3) agents. Three recurrent architectures are evaluated under controlled disturbances with different temporal characteristics. Results show that action history is particularly important when observed responses depend on previous actions, and that processing past and current action-observation information within a unified temporal sequence improves performance compared with using separate branches. We also introduce H-TD3, which reuses recurrent states generated by the actor to initialize the critic, reducing duplicated sequence processing. The architectures are further tested in a simulation-based differential-drive rover motion-regulation task under hidden asymmetric wheel slip. Recurrent architectures retain their advantage under the physically motivated multiplicative wheel-slip model, while policies trained with abstract temporally structured disturbances transfer more effectively to previously unseen wheel-slip dynamics than policies trained without disturbances. These findings provide practical guidance for recurrent RL under partial observability and evolving hidden disturbances.

23 pages, 15 figures, 5 tables. Substantially revised and extended from v2 with a new simulation-based differential-drive rover case study under hidden asymmetric wheel slip, additional transfer and robustness evaluations, revised framing, and an added coauthor. Previously titled "Dynamic Deep-Reinforcement-Learning Algorithm in Partially Observable Markov Decision Processes."