83 citations · 532 across the 70 of their papers we have counts for
71 papers
Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning
Matthew Schlegel, Volodymyr Tkachuk, Adam White +1
Building and maintaining state to learn policies and value functions is critical for deploying reinforcement learning (RL) agents in the real world. Recurrent neural networks (RNNs…
Measuring and Mitigating Interference in Reinforcement Learning
Vincent Liu, Han Wang, Ruo Yu Tao +3
Catastrophic interference is common in many network-based learning systems, and many proposals exist for mitigating it. Before overcoming interference we must understand it better.…
Coagent Networks: Generalized and Scaled
James E. Kostas, Scott M. Jordan, Yash Chandak +5
Coagent networks for reinforcement learning (RL) [Thomas and Barto, 2011] provide a powerful and flexible framework for deriving principled learning rules for arbitrary stochastic…
Empirical Design in Reinforcement Learning
Andrew Patterson, Samuel Neumann, Martha White +1
Empirical design in reinforcement learning is no small task. Running good experiments requires attention to detail and at times significant computational resources. While compute r…
Asymptotically Unbiased Off-Policy Policy Evaluation when Reusing Old Data in Nonstationary Environments
Vincent Liu, Yash Chandak, Philip Thomas +1
In this work, we consider the off-policy policy evaluation problem for contextual bandits and finite horizon reinforcement learning in the nonstationary setting. Reusing old data i…
The In-Sample Softmax for Offline Reinforcement Learning
Chenjun Xiao, Han Wang, Yangchen Pan +2
Reinforcement learning (RL) agents can leverage batches of previously collected data to extract a reasonable control policy. An emerging issue in this offline RL setting, however,…
Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networks
Khurram Javed, Haseeb Shah, Rich Sutton +1
Constructing states from sequences of observations is an important component of reinforcement learning agents. One solution for state construction is to use recurrent neural networ…
Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence
Lingwei Zhu, Zheng Chen, Matthew Schlegel +1
Many policy optimization approaches in reinforcement learning incorporate a Kullback-Leilbler (KL) divergence to the previous policy, to prevent the policy from changing too quickl…
Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learning
Brett Daley, Martha White, Christopher Amato +1
Off-policy learning from multistep returns is crucial for sample-efficient reinforcement learning, but counteracting off-policy bias without exacerbating variance is challenging. C…
Goal-Space Planning with Subgoal Models
Chunlok Lo, Kevin Roice, Parham Mohammad Panahi +5
This paper investigates a new approach to model-based reinforcement learning using background planning: mixing (approximate) dynamic programming updates and model-free updates, sim…
No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL
Han Wang, Archit Sakhadeo, Adam White +7
The performance of reinforcement learning (RL) agents is sensitive to the choice of hyperparameters. In real-world settings like robotics or industrial control systems, however, te…
Robust Losses for Learning Value Functions
Andrew Patterson, Victor Liao, Martha White
Most value function learning algorithms in reinforcement learning are based on the mean squared (projected) Bellman error. However, squared errors are known to be sensitive to outl…
Resonance in Weight Space: Covariate Shift Can Drive Divergence of SGD with Momentum
Kirby Banman, Liam Peet-Pare, Nidhi Hegde +2
Most convergence guarantees for stochastic gradient descent with momentum (SGDm) rely on iid sampling. Yet, SGDm is often used outside this regime, in settings with temporally corr…
Investigating the Properties of Neural Network Representations in Reinforcement Learning
Han Wang, Erfan Miahi, Martha White +5
In this paper we investigate the properties of representations learned by deep reinforcement learning systems. Much of the early work on representations for reinforcement learning…
Continual Auxiliary Task Learning
Matthew McLeod, Chunlok Lo, Matthew Schlegel +4
Learning auxiliary tasks, such as multiple predictions about the world, can provide many benefits to reinforcement learning systems. A variety of off-policy learning algorithms hav…
A Temporal-Difference Approach to Policy Gradient Estimation
Samuele Tosatto, Andrew Patterson, Martha White +1
The policy gradient theorem (Sutton et al., 2000) prescribes the usage of a cumulative discounted state distribution under the target policy to approximate the gradient. Most algor…
Quintessential inflation and nonlinear effects of the tachyonic trap mechanism
Mindaugas Karčiauskas, Stanislav Rusak, Alejandro Saez
With the help of the tachyonic trapping mechanism one can potentially solve a number of problems affecting quintessential inflation models. In this mechanism we introduce a trappin…
Representation Alignment in Neural Networks
Ehsan Imani, Wei Hu, Martha White
It is now a standard for neural network representations to be trained on large, publicly available datasets, and used for new problems. The reasons for why neural network represent…
Off-Policy Actor-Critic with Emphatic Weightings
Eric Graves, Ehsan Imani, Raksha Kumaraswamy +1
A variety of theoretically-sound policy gradient algorithms exist for the on-policy setting due to the policy gradient theorem, which provides a simplified form for the gradient. T…
Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning
Vincent Liu, James R. Wright, Martha White
Offline reinforcement learning -- learning a policy from a batch of data -- is known to be hard for general MDPs. These results motivate the need to look at specific classes of MDP…
Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
Alan Chan, Hugo Silva, Sungsu Lim +3
Approximate Policy Iteration (API) algorithms alternate between (approximate) policy evaluation and (approximate) greedification. Many different approaches have been explored for a…
Predictive Representation Learning for Language Modeling
Qingfeng Lan, Luke Kumar, Martha White +1
To effectively perform the task of next-word prediction, long short-term memory networks (LSTMs) must keep track of many types of information. Some information is directly related…
A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning
Andrew Patterson, Adam White, Martha White
Many reinforcement learning algorithms rely on value estimation, however, the most widely used algorithms -- namely temporal difference algorithms -- can diverge under both off-pol…
Scalable Online Recurrent Learning Using Columnar Neural Networks
Khurram Javed, Martha White, Rich Sutton
Structural credit assignment for recurrent learning is challenging. An algorithm called RTRL can compute gradients for recurrent networks online but is computationally intractable…