6 citations · 17 across the 29 of their papers we have counts for
29 papers
Proximal Policy Optimization for Amortized Discrete Sampling
Anna Zykova-Myzina, Timofei Gritsaev, Daniil Tiapkin +1
This paper explores policy gradient algorithms for training stochastic policies to sample from structured discrete probability distributions under the Generative Flow Network (GFlo…
Refined Analysis of Entropy-Regularized Actor-Critic
Safwan Labbi, Paul Mangold, Daniil Tiapkin +1
In this paper, we study the role of the critic in actor--critic for entropy-regularized, finite, discounted environments. We establish that, when the critic is exact, using the lat…
Learning Shortest Paths with Generative Flow Networks
Nikita Morozov, Ian Maksimov, Daniil Tiapkin +1
In this paper, we present a novel learning framework for finding shortest paths in graphs utilizing Generative Flow Networks (GFlowNets). First, we examine theoretical properties o…
Beyond Softmax and Entropy: Convergence Rates of Policy Gradients with f-SoftArgmax Parameterization & Coupled Regularization
Safwan Labbi, Daniil Tiapkin, Paul Mangold +1
Policy gradient methods are known to be highly sensitive to the choice of policy parameterization. In particular, the widely used softmax parameterization can induce ill-conditione…
gfnx: Fast and Scalable Library for Generative Flow Networks in JAX
Daniil Tiapkin, Artem Agarkov, Nikita Morozov +4
In this paper, we present gfnx, a fast and scalable package for training and evaluating Generative Flow Networks (GFlowNets) written in JAX. gfnx provides an extensive set of envir…
Adaptive Destruction Processes for Diffusion Samplers
Timofei Gritsaev, Nikita Morozov, Kirill Tamogashev +5
This paper explores the challenges and benefits of a trainable destruction process in diffusion samplers -- diffusion-based generative models trained to sample an unnormalised dens…
Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games
Antonio Ocello, Daniil Tiapkin, Lorenzo Mancini +2
We introduce Mean-Field Trust Region Policy Optimization (MF-TRPO), a novel algorithm designed to compute approximate Nash equilibria for ergodic Mean-Field Games (MFG) in finite s…
On Global Convergence Rates for Federated Softmax Policy Gradient under Heterogeneous Environments
Safwan Labbi, Paul Mangold, Daniil Tiapkin +1
We provide global convergence rates for vanilla and entropy-regularized federated softmax stochastic policy gradient (FedPG) with local training. We show that FedPG converges to a…
Proximal Point Nash Learning from Human Feedback
Daniil Tiapkin, Daniele Calandriello, Denis Belomestny +5
Traditional Reinforcement Learning from Human Feedback (RLHF) often relies on reward models, frequently assuming preference structures like the Bradley--Terry model, which may not…
On Teacher Hacking in Language Model Distillation
Daniil Tiapkin, Daniele Calandriello, Johan Ferret +4
Post-training of language models (LMs) increasingly relies on the following two stages: (i) knowledge distillation, where the LM is trained to imitate a larger teacher LM, and (ii)…
Revisiting Non-Acyclic GFlowNets in Discrete Environments
Nikita Morozov, Ian Maksimov, Daniil Tiapkin +1
Generative Flow Networks (GFlowNets) are a family of generative models that learn to sample objects from a given probability distribution, potentially known up to a normalizing con…
Federated UCBVI: Communication-Efficient Federated Regret Minimization with Heterogeneous Agents
Safwan Labbi, Daniil Tiapkin, Lorenzo Mancini +2
In this paper, we present the Federated Upper Confidence Bound Value Iteration algorithm (), a novel extension of the algorithm (Azar et al., 2…
Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization
Timofei Gritsaev, Nikita Morozov, Sergey Samsonov +1
Generative Flow Networks (GFlowNets) are a family of generative models that learn to sample objects with probabilities proportional to a given reward function. The key concept behi…
A New Bound on the Cumulant Generating Function of Dirichlet Processes
Pierre Perrault, Denis Belomestny, Pierre Ménard +4
In this paper, we introduce a novel approach for bounding the cumulant generating function (CGF) of a Dirichlet process (DP) , using superadditivity. In par…
Narrowing the Gap between Adversarial and Stochastic MDPs via Policy Optimization
Daniil Tiapkin, Evgenii Chzhen, Gilles Stoltz
We consider the problem of learning in adversarial Markov decision processes [MDPs] with an oblivious adversary in a full-information setting. The agent interacts with an environme…
Improving GFlowNets with Monte Carlo Tree Search
Nikita Morozov, Daniil Tiapkin, Sergey Samsonov +2
Generative Flow Networks (GFlowNets) treat sampling from distributions over compositional discrete spaces as a sequential decision-making problem, training a stochastic policy to c…
Incentivized Learning in Principal-Agent Bandit Games
Antoine Scheid, Daniil Tiapkin, Etienne Boursier +5
This work considers a repeated principal-agent bandit game, where the principal can only interact with her environment through the agent. The principal and the agent have misaligne…
Model-free Posterior Sampling via Learning Rate Randomization
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello +6
In this paper, we introduce Randomized Q-learning (RandQL), a novel randomized model-free algorithm for regret minimization in episodic Markov Decision Processes (MDPs). To the bes…
Demonstration-Regularized RL
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello +5
Incorporating expert demonstrations has empirically helped to improve the sample efficiency of reinforcement learning (RL). This paper quantifies theoretically to what extent this…
Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stability
Sergey Samsonov, Daniil Tiapkin, Alexey Naumov +1
In this paper we consider the problem of obtaining sharp bounds for the performance of temporal difference (TD) methods with linear function approximation for policy evaluation in…
Generative Flow Networks as Entropy-Regularized RL
Daniil Tiapkin, Nikita Morozov, Alexey Naumov +1
The recently proposed generative flow networks (GFlowNets) are a method of training a policy to sample compositional discrete objects with probabilities proportional to a given rew…
Sharp Deviations Bounds for Dirichlet Weighted Sums with Application to analysis of Bayesian algorithms
Denis Belomestny, Pierre Menard, Alexey Naumov +2
In this work, we derive sharp non-asymptotic deviation bounds for weighted sums of Dirichlet random variables. These bounds are based on a novel integral representation of the dens…
Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold
Sholom Schechtman, Daniil Tiapkin, Michael Muehlebach +1
We consider the problem of minimizing a non-convex function over a smooth manifold . We propose a novel algorithm, the Orthogonal Directions Constrained Gradient Metho…
Fast Rates for Maximum Entropy Exploration
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello +7
We address the challenge of exploration in reinforcement learning (RL) when the agent operates in an unknown environment with sparse or no rewards. In this work, we study the maxim…