6 citations · 6 across the 12 of their papers we have counts for
5 papers · 1 filter
SDG-MoE: Signed Debate Graph Mixture-of-Experts
Stepan Kulibaba, Kirill Labzin, Artem Dzhalilov +4
Sparse MoE models achieve a good balance between capacity and compute by routing each token to a small subset of experts. However, in most MoE architectures, once a token is routed…
Quantum-Inspired Episode Selection for Monte Carlo Reinforcement Learning via QUBO Optimization
Hadi Salloum, Ali Jnadi, Yaroslav Kholodov +1
Monte Carlo (MC) reinforcement learning suffers from high sample complexity, especially in environments with sparse rewards, large state spaces, and correlated trajectories. We add…
Managing Self-Learning Experts under Per-Round Budget Constraints
Ilgam Latypov, Alexandra Suvorikova, Alexey Kroshnin +2
This paper addresses the problem of sequential decision-making under learning budget constraints. Such settings naturally arise in applications like managing a portfolio of bandit…
AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models
Nikolay Kutuzov, Makar Baderko, Stepan Kulibaba +4
Scaling distributed training of Large Language Models (LLMs) requires not only algorithmic advances but also efficient utilization of heterogeneous hardware resources. While existi…
Activations and Gradients Compression for Model-Parallel Training
Mikhail Rudakov, Aleksandr Beznosikov, Yaroslav Kholodov +1
Large neural networks require enormous computational clusters of machines. Model-parallel training, when the model architecture is partitioned sequentially between workers, is a po…