6 citations · 31 across the 20 of their papers we have counts for
21 papers
Global Exponential Convergence of Two-Layer Linear Network Training
Stephen Y Zhang, Gabriel Peyré
We prove global exponential (linear) convergence with an explicit rate in the rich scaling for wide two-layer linear networks trained with smooth Polyak-Lojasiewicz predictor losse…
Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models
Nilaksh, Saurav Jha, Artem Zholus +1
World model-based policy evaluation is a practical proxy for testing real-world robot control by rolling out candidate actions in action-conditioned video diffusion models. As thes…
TAPNext++: What's Next for Tracking Any Point (TAP)?
Sebastian Jung, Artem Zholus, Martin Sundermeyer +6
Tracking-Any-Point (TAP) models aim to track any point through a video which is a crucial task in AR/XR and robotics applications. The recently introduced TAPNext approach proposes…
Hierarchical Planning with Latent World Models
Wancong Zhang, Basile Terver, Artem Zholus +8
World models are a promising path to zero-shot embodied control through planning. However, existing world model planners struggle on long-horizon, multi-stage tasks: prediction err…
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
Mido Assran, Adrien Bardes, David Fan +27
A major challenge for modern AI is to learn to understand the world and learn to act largely by observation. This paper explores a self-supervised approach that combines internet-s…
TAPNext: Tracking Any Point (TAP) as Next Token Prediction
Artem Zholus, Carl Doersch, Yi Yang +7
Tracking Any Point (TAP) in a video is a challenging computer vision problem with many demonstrated applications in robotics, video editing, and 3D reconstruction. Existing methods…
TRecViT: A Recurrent Video Transformer
Viorica Pătrăucean, Xu Owen He, Joseph Heyward +10
We propose a novel block for \emph{causal} video modelling. It relies on a time-space-channel factorisation with dedicated blocks for each dimension: gated linear recurrent units (…
Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation
Egor Cherepanov, Nikita Kachaev, Artem Zholus +2
The incorporation of memory into agents is essential for numerous tasks within the domain of Reinforcement Learning (RL). In particular, memory is paramount for tasks that require…
Interpretability in Action: Exploratory Analysis of VPT, a Minecraft Agent
Karolis Jucys, George Adamopoulos, Mehrab Hamidi +7
Understanding the mechanisms behind decisions taken by large foundation models in sequential decision making tasks is critical to ensuring that such systems operate transparently a…
IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents
Shrestha Mohanty, Negar Arabzadeh, Andrea Tupini +5
Seamless interaction between AI agents and humans using natural language remains a key goal in AI research. This paper addresses the challenges of developing interactive agents cap…
BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning
Artem Zholus, Maksim Kuznetsov, Roman Schutski +4
Generating novel active molecules for a given protein is an extremely challenging task for generative models that requires an understanding of the complex physical interactions bet…
Mastering Memory Tasks with World Models
Mohammad Reza Samsami, Artem Zholus, Janarthanan Rajendran +1
Current model-based reinforcement learning (MBRL) agents struggle with long-term dependencies. This limits their ability to effectively solve tasks involving extended time gaps bet…
Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions
Shrestha Mohanty, Negar Arabzadeh, Julia Kiseleva +6
Human intelligence's adaptability is remarkable, allowing us to adjust to new tasks and multi-modal environments swiftly. This skill is evident from a young age as we acquire new a…
Collecting Interactive Multi-modal Datasets for Grounded Language Understanding
Shrestha Mohanty, Negar Arabzadeh, Milagro Teruel +9
Human intelligence can remarkably adapt quickly to new tasks and environments. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by…
Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructions
Alexey Skrynnik, Zoya Volovikova, Marc-Alexandre Côté +9
The adoption of pre-trained language models to generate action plans for embodied agents is a promising research strategy. However, execution of instructions in real or simulated e…
IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agents
Artem Zholus, Alexey Skrynnik, Shrestha Mohanty +5
We present the IGLU Gridworld: a reinforcement learning environment for building and evaluating language conditioned embodied agents in a scalable way. The environment features vis…
IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022
Julia Kiseleva, Alexey Skrynnik, Artem Zholus +14
Human intelligence has the remarkable ability to adapt to new tasks and environments quickly. Starting from a very young age, humans acquire new skills and learn how to solve new t…
Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021
Julia Kiseleva, Ziming Li, Mohammad Aliannejadi +18
Human intelligence has the remarkable ability to quickly adapt to new tasks and environments. Starting from a very young age, humans acquire new skills and learn how to solve new t…
Multitask Adaptation by Retrospective Exploration with Learned World Models
Artem Zholus, Aleksandr I. Panov
Model-based reinforcement learning (MBRL) allows solving complex tasks in a sample-efficient manner. However, no information is reused between the tasks. In this work, we propose a…
NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment
Julia Kiseleva, Ziming Li, Mohammad Aliannejadi +12
Human intelligence has the remarkable ability to adapt to new tasks and environments quickly. Starting from a very young age, humans acquire new skills and learn how to solve new t…
Continuous Histogram Loss: Beyond Neural Similarity
Artem Zholus, Eugene Lane
Similarity learning has gained a lot of attention from researches in recent years and tons of successful approaches have been recently proposed. However, the majority of the state-…