9 papers · 1 filter
V-SHARE: Provable Convergence Guarantees for Advantage-based Algorithms
Kintan Saha, Hamed Masoudi, Xinyi Sheng +3
In reinforcement learning, advantage-based value-learning methods, unlike standard Q-learning, share information across actions by decomposing the action-value function into a stat…
SUN: Reaching for Novelty in Reinforcement Learning
Wenyan Yang, Arsenii Mustafin, Dominik Baumann +2
Exploration in reinforcement learning (RL) remains a fundamental challenge. Recent goal-conditioned RL strategies (which select goals to encourage broader state coverage) have show…
Ergodicity in reinforcement learning
Dominik Baumann, Erfaun Noorani, Arsenii Mustafin +5
In reinforcement learning, we typically aim to optimize the expected value of the sum of rewards an agent collects over a trajectory. However, if the process generating these rewar…
Revisiting Value Iteration: Unified Analysis of Discounted and Average-Reward Cases
Arsenii Mustafin, Xinyi Sheng, Dominik Baumann
While Value Iteration (VI) is one of the most fundamental algorithms in Reinforcement Learning, its theoretical convergence guarantees still exhibit a persistent mismatch with empi…
Geometric Re-Analysis of Classical MDP Solving Algorithms
Arsenii Mustafin, Aleksei Pakharev, Alex Olshevsky +1
We build on a recently introduced geometric interpretation of Markov Decision Processes (MDPs) to analyze classical MDP-solving algorithms: Value Iteration (VI) and Policy Iteratio…
Analysis of Value Iteration Through Absolute Probability Sequences
Arsenii Mustafin, Sebastien Colla, Alex Olshevsky +1
Value Iteration is a widely used algorithm for solving Markov Decision Processes (MDPs). While previous studies have extensively analyzed its convergence properties, they primarily…
MDP Geometry, Normalization and Reward Balancing Solvers
Arsenii Mustafin, Aleksei Pakharev, Alex Olshevsky +1
We present a new geometric interpretation of Markov Decision Processes (MDPs) with a natural normalization procedure that allows us to adjust the value function at each state witho…
On Value Iteration Convergence in Connected MDPs
Arsenii Mustafin, Alex Olshevsky, Ioannis Ch. Paschalidis
This paper establishes that an MDP with a unique optimal policy and ergodic associated transition matrix ensures the convergence of various versions of the Value Iteration algorith…
Closing the gap between SVRG and TD-SVRG with Gradient Splitting
Arsenii Mustafin, Alex Olshevsky, Ioannis Ch. Paschalidis
Temporal difference (TD) learning is a policy evaluation in reinforcement learning whose performance can be enhanced by variance reduction methods. Recently, multiple works have so…