Publications (46)
Mamba-Adaptor: State Space Model Adaptor for Visual Recognition
Fei Xie, Jiahao Nie, Yujin Tang +2
Recent State Space Models (SSM), especially Mamba, have demonstrated impressive performance in visual modeling and possess superior model efficiency. However, the application of Ma…
Large Language Models to Diffusion Finetuning
Edoardo Cetin, Tianyu Zhao, Yujin Tang
We propose a new finetuning method to provide pre-trained large language models (LMs) the ability to scale test-time compute through the diffusion framework. By increasing the numb…
Discovering Novel LLM Experts via Task-Capability Coevolution
Andrew Dai, Boris Meinardus, Ciaran Regan +2
Frontier model developers aim to train models continually to possess emergent, diverse capabilities. To extend capabilities, the current pre-training and post-training paradigm req…
PostRainBench: A comprehensive benchmark and a new model for precipitation forecasting
Yujin Tang, Jiaming Zhou, Xiang Pan +2
Accurate precipitation forecasting is a vital challenge of societal importance. Though data-driven approaches have emerged as a widely used solution, solely relying on data-driven…
Evolution Transformer: In-Context Evolutionary Optimization
Robert Tjarko Lange, Yingtao Tian, Yujin Tang
Evolutionary optimization algorithms are often derived from loose biological analogies and struggle to leverage information obtained during the sequential course of optimization. A…
PatchMixer: A Patch-Mixing Architecture for Long-Term Time Series Forecasting
Zeying Gong, Yujin Tang, Junwei Liang
Although the Transformer has been the dominant architecture for time series forecasting tasks in recent years, a fundamental challenge remains: the permutation-invariant self-atten…
Learning Agile Locomotion via Adversarial Training
Yujin Tang, Jie Tan, Tatsuya Harada
Developing controllers for agile locomotion is a long-standing challenge for legged robots. Reinforcement learning (RL) and Evolution Strategy (ES) hold the promise of automating t…
Collective Intelligence for 2D Push Manipulations with Mobile Robots
So Kuroki, Tatsuya Matsushima, Jumpei Arima +4
While natural systems often present collective intelligence that allows them to self-organize and adapt to changes, the equivalent is missing in most artificial systems. We explore…
KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI
So Kuroki, Yotaro Kubo, Takuya Akiba +1
Real-time speech-to-speech (S2S) models excel at generating natural, low-latency conversational responses but often lack deep knowledge and semantic understanding. Conversely, casc…
Position: Leverage Foundational Models for Black-Box Optimization
Xingyou Song, Yingtao Tian, Robert Tjarko Lange +3
Undeniably, Large Language Models (LLMs) have stirred an extraordinary wave of innovation in the machine learning research domain, resulting in substantial impact across diverse fi…
Digital Red Queen: Adversarial Program Evolution in Core War with LLMs
Akarsh Kumar, Ryan Bahlous-Boldi, Prafull Sharma +4
Large language models (LLMs) are increasingly being used to evolve solutions to problems in many domains, in a process inspired by biological evolution. However, unlike biological…
Video Prediction Transformers without Recurrence or Convolution
Yujin Tang, Lu Qi, Xiangtai Li +2
Video prediction has witnessed the emergence of RNN-based models led by ConvLSTM, and CNN-based models led by SimVP. Following the significant success of ViT, recent works have int…
Competition and Attraction Improve Model Fusion
João Abrantes, Robert Tjarko Lange, Yujin Tang
Model merging is a powerful technique for integrating the specialized knowledge of multiple machine learning models into a single model. However, existing methods require manually…
Large Language Models As Evolution Strategies
Robert Tjarko Lange, Yingtao Tian, Yujin Tang
Large Transformer models are capable of implementing a plethora of so-called in-context learning algorithms. These include gradient descent, classification, sequence completion, tr…
Shachi: A Modular, Controllable Framework for LLM-Based Agent-Based Modeling of Emergent Collective Behavior
So Kuroki, Yingtao Tian, Kou Misaki +3
How collective behaviors emerge from the interactions of individual LLM-driven agents is a central question in artificial life, yet controlled study of these emergent dynamics has…
Evolutionary Optimization of Model Merging Recipes
Takuya Akiba, Makoto Shing, Yujin Tang +2
Large language models (LLMs) have become increasingly capable, but their development often requires substantial computational resources. While model merging has emerged as a cost-e…
Evolution of Collective AI Beyond Individual Optimization
Ryosuke Takata, Yujin Tang, Yingtao Tian +3
This study investigates collective behaviors that emerge from a group of homogeneous individuals optimized for a specific capability. We created a group of simple, identical neural…
Sakana Fugu Technical Report
Yujin Tang, Edoardo Cetin, Jinglue Xu +11
The capabilities of frontier Large Language Models (LLMs) continue to advance, with different providers increasingly specializing in distinct domains. This raises a natural next ob…
The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning
Yujin Tang, David Ha
In complex systems, we often observe complex global behavior emerge from a collection of agents interacting with each other in their environment, with each individual agent acting…
TRINITY: An Evolved LLM Coordinator
Jinglue Xu, Qi Sun, Peter Schwendeman +3
Combining diverse foundation models is promising, but weight-merging is limited by mismatched architectures and closed APIs. Trinity addresses this with a lightweight coordinator t…
Learning to Orchestrate Agents in Natural Language with the Conductor
Stefan Nielsen, Edoardo Cetin, Peter Schwendeman +3
Powerful large language models (LLMs) from different providers have been expensively trained and finetuned to specialize across varying domains. In this work, we introduce a new ki…
Text-to-LoRA: Instant Transformer Adaption
Rujikorn Charakorn, Edoardo Cetin, Yujin Tang +1
While Foundation Models provide a general tool for rapid content creation, they regularly require task-specific adaptation. Traditionally, this exercise involves careful curation o…
Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning
Can Jin, Hongwu Peng, Qixin Zhang +3
Multi-agent systems (MAS) built on large language models (LLMs) offer a promising path toward solving complex, real-world tasks that single-agent systems often struggle to manage.…
Automating the Search for Artificial Life with Foundation Models
Akarsh Kumar, Chris Lu, Louis Kirsch +4
With the recent Nobel Prize awarded for radical advances in protein discovery, foundation models (FMs) for exploring large combinatorial spaces promise to revolutionize many scient…
NeuroEvoBench: Benchmarking Evolutionary Optimizers for Deep Learning Applications
Robert Tjarko Lange, Yujin Tang, Yingtao Tian
Recently, the Deep Learning community has become interested in evolutionary optimization (EO) as a means to address hard optimization problems, e.g. meta-learning through long inne…
Learning Video Dynamics with Predictive Differentiable Rendering
Yujin Tang, Tian Zhou, Xin Lin +5
How to accurately predict a high-fidelity future world? While the visual world is inherently continuous, existing deterministic video prediction models operate in discrete pixel sp…
Neuroevolution of Self-Interpretable Agents
Yujin Tang, Duong Nguyen, David Ha
Inattentional blindness is the psychological phenomenon that causes one to miss things in plain sight. It is a consequence of the selective attention in perception that lets us rem…
Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization
Robert Tjarko Lange, Qi Sun, Aaditya Prasad +3
Recent advances in large language models (LLMs) demonstrate their effectiveness in scaling test-time compute for software engineering tasks. However, these approaches often focus o…
An Evolved Universal Transformer Memory
Edoardo Cetin, Qi Sun, Tianyu Zhao +1
Prior methods propose to offset the escalating costs of modern foundation models by dropping specific parts of their contexts with hand-designed rules, while attempting to preserve…
Collective Intelligence for Deep Learning: A Survey of Recent Developments
David Ha, Yujin Tang
In the past decade, we have witnessed the rise of deep learning to dominate the field of artificial intelligence. Advances in artificial neural networks alongside corresponding adv…
DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards
Shanchuan Wan, Yujin Tang, Yingtao Tian +1
Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness is a deciding factor in the performance of RL algorithms, especially when facing sparse ex…
SayTap: Language to Quadrupedal Locomotion
Yujin Tang, Wenhao Yu, Jie Tan +3
Large language models (LLMs) have demonstrated the potential to perform high-level planning. Yet, it remains a challenge for LLMs to comprehend low-level commands, such as joint an…
SAIL: Test-Time Scaling for In-Context Imitation Learning with VLM
Makoto Sato, Yusuke Iwasawa, Yujin Tang +1
In-context imitation learning allows robots to acquire skills from demonstrations, yet one-shot trajectory generation remains fragile under environmental variation. We propose SAIL…
Recursive Harness Self-Improvement
Hyunin Lee, Jinglue Xu, Jeffrey Seely +3
Under model--harness co-evolution, harnesses are not merely inference-time scaffolds but data-generating components whose execution traces can shape future foundation models. This…
Bridging Past and Future: Distribution-Aware Alignment for Time Series Forecasting
Yifan Hu, Jie Yang, Tian Zhou +4
Although contrastive and other representation-learning methods have long been explored in vision and NLP, their adoption in modern time series forecasters remains limited. We belie…
Transformer-Squared: Self-adaptive LLMs
Qi Sun, Edoardo Cetin, Yujin Tang
Self-adaptive large language models (LLMs) aim to solve the challenges posed by traditional fine-tuning methods, which are often computationally intensive and static in their abili…
Learning to Generalize with Object-centric Agents in the Open World Survival Game Crafter
Aleksandar StaniÄ, Yujin Tang, David Ha +1
Reinforcement learning agents must generalize beyond their training experience. Prior work has focused mostly on identical training and evaluation environments. Starting from the r…
Open X-Embodiment: Robotic Learning Datasets and RT-X Models
Embodiment Collaboration, Abby O'Neill, Abdul Rehman +291
Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, thi…
Evolutionary Context Search for Automated Skill Acquisition
Qi Sun, Stefan Nielsen, Rio Yokota +1
Large Language Models cannot reliably acquire new knowledge post-deployment -- even when relevant text resources exist, models fail to transform them into actionable knowledge with…
Agent Skill Acquisition for Large Language Models via CycleQD
So Kuroki, Taishi Nakamura, Takuya Akiba +1
Training large language models to acquire specific skills remains a challenging endeavor. Conventional training approaches often struggle with data distribution imbalances and inad…
Evolving Modular Soft Robots without Explicit Inter-Module Communication using Local Self-Attention
Federico Pigozzi, Yujin Tang, Eric Medvet +1
Modularity in robotics holds great potential. In principle, modular robots can be disassembled and reassembled in different robots, and possibly perform new tasks. Nevertheless, ac…
VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting
Yujin Tang, Peijie Dong, Zhenheng Tang +2
Combining CNNs or ViTs, with RNNs for spatiotemporal forecasting, has yielded unparalleled results in predicting temporal and spatial dynamics. However, modeling extensive global i…
EvoJAX: Hardware-Accelerated Neuroevolution
Yujin Tang, Yingtao Tian, David Ha
Evolutionary computation has been shown to be a highly effective method for training neural networks, particularly when employed at scale on CPU clusters. Recent work have also sho…
Reinforcement Learning Teachers of Test Time Scaling
Edoardo Cetin, Tianyu Zhao, Yujin Tang
Training reasoning language models (LMs) with reinforcement learning (RL) for one-hot correctness inherently relies on the LM being able to explore and solve its task with some cha…
DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection
Yujin Tang, Chenming Shang, Ruize Xu +1
Research on agent memory has matured rapidly, but almost entirely on the text side: few existing benchmarks ask, in an interactive environment, when an agent genuinely needs to rem…
LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views
Yuji Roh, Qingyun Liu, Huan Gui +8
Fine-tuning is becoming widely used for leveraging the power of pre-trained foundation models in new downstream tasks. While there are many successes of fine-tuning on various task…