9 citations · 13 across the 13 of their papers we have counts for
14 papers
Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation
Ku Onoda, Paavo Parmas, Hiroki Furuta +4
Text-to-image (T2I) models can synthesize realistic, prompt-aligned images, yet samples generated for the same prompt often cover only a small subset of visually distinct modes. Th…
OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation
Paavo Parmas, Yongmin Kim, Kohsei Matsutani +5
Policy-gradient methods usually optimize expected return, but many real world applications care about distributional properties of returns: tail risk, outlier robustness, or best-o…
On Advantage Estimates for Max@K Policy Gradients
Shota Takashiro, Soichiro Nishimori, Paavo Parmas +6
Reinforcement learning with verifiable rewards is widely used for post-training reasoning models, but sparse outcome rewards make exploration difficult. A complementary approach is…
Retry Policy Gradients in Continuous Action Spaces
Soichiro Nishimori, Paavo Parmas
Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration wi…
Regret Analysis of Retry-Based Bandits
Bingkui Tong, Junpei Komiyama, Soichiro Nishimori +1
We provide the first regret analysis of ReMax in stochastic multi-armed bandits. Originally introduced for reinforcement learning, ReMax is motivated by the role of exploration whe…
Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada +4
In reinforcement learning (RL), agents benefit from exploration only because they repeatedly encounter similar states: trying different actions can improve performance or reduce un…
Does "Do Differentiable Simulators Give Better Policy Gradients?'' Give Better Policy Gradients?
Ku Onoda, Paavo Parmas, Manato Yaguchi +1
In policy gradient reinforcement learning, access to a differentiable model enables 1st-order gradient estimation that accelerates learning compared to relying solely on derivative…
Double Horizon Model-Based Policy Optimization
Akihiro Kubo, Paavo Parmas, Shin Ishii
Model-based reinforcement learning (MBRL) reduces the cost of real-environment sampling by generating synthetic trajectories (called rollouts) from a learned dynamics model. Howeve…
Bi-Level Reinforcement Learning Pathway for Sim-to-Real Optimality
Akhil S Anand, Shambhuraj Sawant, Paavo Parmas +3
Training Reinforcement Learning (RL) policies using simulation models before deployment in real-world environments is a common strategy when real-world interaction is expensive. Th…
Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form
Toshinori Kitamura, Tadashi Kozuno, Wataru Kumagai +6
Designing a safe policy for uncertain environments is crucial in real-world control systems. However, this challenge remains inadequately addressed within the Markov decision proce…
A unified view of likelihood ratio and reparameterization gradients
Paavo Parmas, Masashi Sugiyama
Reparameterization (RP) and likelihood ratio (LR) gradient estimators are used to estimate gradients of expectations throughout machine learning and reinforcement learning; however…
A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
Paavo Parmas, Masashi Sugiyama
Reparameterization (RP) and likelihood ratio (LR) gradient estimators are used throughout machine and reinforcement learning; however, they are usually explained as simple mathemat…
Total stochastic gradient algorithms and applications in reinforcement learning
Paavo Parmas
Backpropagation and the chain rule of derivatives have been prominent; however, the total derivative rule has not enjoyed the same amount of attention. In this work we show how the…
PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos
Paavo Parmas, Carl Edward Rasmussen, Jan Peters +1
Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instabil…