6 papers · 1 filter
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Damien Teney, Liangze Jiang, Hemanth Saratchandran +1
Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be…
Meta-RL Induces Exploration in Language Agents
Yulun Jiang, Liangze Jiang, Damien Teney +2
Reinforcement learning (RL) has enabled the training of large language model (LLM) agents to interact with the environment and to solve multi-turn long-horizon tasks. However, the…
Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran +2
Pretraining on large, semantically rich datasets is key for developing language models. Surprisingly, recent studies have shown that even synthetic data, generated procedurally thr…
Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild
Damien Teney, Liangze Jiang, Florin Gogianu +1
Neural architectures tend to fit their data with relatively simple functions. This "simplicity bias" is widely regarded as key to their success. This paper explores the limits of t…
OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?
Liangze Jiang, Damien Teney
Out-of-distribution (OOD) generalization is challenging because distribution shifts come in many forms. Numerous algorithms exist to address specific settings, but choosing the rig…
Unraveling the Key Components of OOD Generalization via Diversification
Harold Benoit, Liangze Jiang, Andrei Atanov +3
Supervised learning datasets may contain multiple cues that explain the training set equally well, i.e., learning any of them would lead to the correct predictions on the training…