261 citations · 331 across the 17 of their papers we have counts for
17 papers
Extending LLM Context via Associative Recurrent Memory
Gleb Kuzmin, Ivan Rodkin, Aydar Bulatov +8
Extending the context length of large language models (LLMs) is critical for many real-world applications, yet standard transformers remain constrained by quadratic compute and lin…
Learning to Communicate Locally for Large-Scale Multi-Agent Pathfinding
Valeriy Vyaltsev, Alsu Sagirova, Anton Andreychuk +5
Multi-agent pathfinding (MAPF) is a widely used abstraction for multi-robot trajectory planning problems, where multiple homogeneous agents move simultaneously within a shared envi…
Revisiting Tree Search for LLMs: Gumbel and Sequential Halving for Budget-Scalable Reasoning
Leonid Ugadiarov, Yuri Kuratov, Aleksandr Panov +1
Neural tree search is a powerful decision-making algorithm widely used in complex domains such as game playing and model-based reinforcement learning. Recent work has applied Alpha…
GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent
Yuri Kuratov, Matvey Kairov, Aydar Bulatov +2
Many large language model applications require conditioning on long contexts. Transformers typically support this by storing a large per-layer KV-cache of past activations, which i…
Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling
Ivan Rodkin, Daniil Orel, Konstantin Smirnov +9
Reasoning is a core capability of large language models, yet how multi-step reasoning is learned and executed remains unclear. We study this question in a controlled cellular-autom…
Diagonal Batching Unlocks Parallelism in Recurrent Memory Transformers for Long Contexts
Danil Sivtsov, Ivan Rodkin, Gleb Kuzmin +2
Transformer models struggle with long-context inference due to their quadratic time and linear memory complexity. Recurrent Memory Transformers (RMTs) offer a solution by reducing…
Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity
Yuri Kuratov, Mikhail Arkhipov, Aydar Bulatov +1
A range of recent works addresses the problem of compression of sequence of tokens into a shorter sequence of real-valued vectors to be used as inputs instead of token embeddings o…
SRMT: Shared Memory for Multi-agent Lifelong Pathfinding
Alsu Sagirova, Yuri Kuratov, Mikhail Burtsev
Coordination in decentralized multi-agent reinforcement learning (MARL) necessitates that agents share information about their behavior and intentions. Existing approaches rely on…
Long Input Benchmark for Russian Analysis
Igor Churin, Murat Apishev, Maria Tikhonova +5
Recent advancements in Natural Language Processing (NLP) have fostered the development of Large Language Models (LLMs) that can solve an immense variety of tasks. One of the key as…
Associative Recurrent Memory Transformer
Ivan Rodkin, Yuri Kuratov, Aydar Bulatov +1
This paper addresses the challenge of creating a neural architecture for very long sequences that requires constant time for processing new information at each time step. Our appro…
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Yuri Kuratov, Aydar Bulatov, Petr Anokhin +4
In recent years, the input context sizes of large language models (LLMs) have increased dramatically. However, existing evaluation methods have not kept pace, failing to comprehens…
In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss
Yuri Kuratov, Aydar Bulatov, Petr Anokhin +3
This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark design…
Scaling Transformer to 1M tokens and beyond with RMT
Aydar Bulatov, Yuri Kuratov, Yermek Kapushev +1
A major limitation for the broader scope of problems solvable by transformers is the quadratic scaling of computational complexity with input size. In this study, we investigate th…
Recurrent Memory Transformer
Aydar Bulatov, Yuri Kuratov, Mikhail S. Burtsev
Transformer-based models show their effectiveness across multiple domains and tasks. The self-attention allows to combine information from all sequence elements into context-aware…
Knowledge Distillation of Russian Language Models with Reduction of Vocabulary
Alina Kolesnikova, Yuri Kuratov, Vasily Konovalov +1
Today, transformer language models serve as a core component for majority of natural language processing tasks. Industrial application of such models requires minimization of compu…
Goal-Oriented Multi-Task BERT-Based Dialogue State Tracker
Pavel Gulyaev, Eugenia Elistratova, Vasily Konovalov +3
Dialogue State Tracking (DST) is a core component of virtual assistants such as Alexa or Siri. To accomplish various tasks, these assistants need to support an increasing number of…
Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
Yuri Kuratov, Mikhail Arkhipov
The paper introduces methods of adaptation of multilingual masked language models for a specific language. Pre-trained bidirectional language models show state-of-the-art performan…