130 citations · 297 across the 16 of their papers we have counts for
18 papers
SIMA 2: A Generalist Embodied Agent for Virtual Worlds
SIMA team, Adrian Bolton, Alexander Lerchner +63
We introduce SIMA 2, a generalist embodied agent that understands and acts in a wide variety of 3D virtual worlds. Built upon a Gemini foundation model, SIMA 2 represents a signifi…
Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiences
Andrew Kyle Lampinen, Martin Engelcke, Yuxuan Li +2
When do machine learning systems fail to generalize, and what mechanisms could improve their generalization? Here, we draw inspiration from cognitive science to argue that one weak…
Can foundation models actively gather information in interactive environments to test hypotheses?
Danny P. Sawyer, Nan Rosemary Ke, Hubert Soyer +9
Foundation models excel at single-turn reasoning but struggle with multi-turn exploration in dynamic environments, a requirement for many real-world challenges. We evaluated these…
Scaling Instructable Agents Across Many Simulated Worlds
SIMA Team, Maria Abi Raad, Arun Ahuja +91
Building embodied AI systems that can follow arbitrary language instructions in any 3D environment is a key challenge for creating general AI. Accomplishing this goal requires lear…
Reaching Through Latent Space: From Joint Statistics to Path Planning in Manipulation
Chia-Man Hung, Shaohong Zhong, Walter Goodwin +4
We present a novel approach to path planning for robotic manipulators, in which paths are produced via iterative optimisation in the latent space of a generative model of robot pos…
VAE-Loco: Versatile Quadruped Locomotion by Learning a Disentangled Gait Representation
Alexander L. Mitchell, Wolfgang Merkt, Mathieu Geisert +5
Quadruped locomotion is rapidly maturing to a degree where robots are able to realise highly dynamic manoeuvres. However, current planners are unable to vary key gait parameters of…
Next Steps: Learning a Disentangled Gait Representation for Versatile Quadruped Locomotion
Alexander L. Mitchell, Wolfgang Merkt, Mathieu Geisert +5
Quadruped locomotion is rapidly maturing to a degree where robots now routinely traverse a variety of unstructured terrains. However, while gaits can be varied typically by selecti…
Universal Approximation of Functions on Sets
Edward Wagstaff, Fabian B. Fuchs, Martin Engelcke +2
Modelling functions of sets, or equivalently, permutation-invariant functions, is a long-standing challenge in machine learning. Deep Sets is a popular method which is known to be…
APEX: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation
Yizhe Wu, Oiwi Parker Jones, Martin Engelcke +1
Recent advances in unsupervised learning for object detection, segmentation, and tracking hold significant promise for applications in robotics. A common approach is to frame these…
GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
Martin Engelcke, Oiwi Parker Jones, Ingmar Posner
Advances in unsupervised learning of object-representations have culminated in the development of a broad range of methods for unsupervised object segmentation and interpretable ob…
Reconstruction Bottlenecks in Object-Centric Generative Models
Martin Engelcke, Oiwi Parker Jones, Ingmar Posner
A range of methods with suitable inductive biases exist to learn interpretable object-centric representations of images without supervision. However, these are largely restricted t…
First Steps: Latent-Space Control with Semantic Constraints for Quadruped Locomotion
Alexander L. Mitchell, Martin Engelcke, Oiwi Parker Jones +5
Traditional approaches to quadruped control frequently employ simplified, hand-derived models. This significantly reduces the capability of the robot since its effective kinematic…
RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
Sebastien Ehrhardt, Oliver Groth, Aron Monszpart +4
We present RELATE, a model that learns to generate physically plausible scenes and videos of multiple interacting objects. Similar to other generative approaches, RELATE is trained…
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones +1
Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects…
On the Limitations of Representing Functions on Sets
Edward Wagstaff, Fabian B. Fuchs, Martin Engelcke +2
Recent work on the representation of functions on sets has considered the use of summation in a latent space to enforce permutation invariance. In particular, it has been conjectur…
3D Semantic Segmentation with Submanifold Sparse Convolutional Networks
Benjamin Graham, Martin Engelcke, Laurens van der Maaten
Convolutional networks are the de-facto standard for analyzing spatio-temporal data such as images, videos, and 3D shapes. Whilst some of this data is naturally dense (e.g., photos…
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
Li Yi, Lin Shao, Manolis Savva +47
We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database. The benchmark consists of two tasks: part-level segmentation…
Vote3Deep: Fast Object Detection in 3D Point Clouds Using Efficient Convolutional Neural Networks
Martin Engelcke, Dushyant Rao, Dominic Zeng Wang +2
This paper proposes a computationally efficient approach to detecting objects natively in 3D point clouds using convolutional neural networks (CNNs). In particular, this is achieve…