4 papers
On the Limitations and Capabilities of Position Embeddings for Length Generalization
Yang Chen, Yitao Liang, Zhouchen Lin
In Transformers, Position Embeddings (PEs) significantly influence Length Generalization (LG) performance, yet their fundamental role remains unclear. In this work, we investigate…
Low-Dimension-to-High-Dimension Generalization And Its Implications for Length Generalization
Yang Chen, Long Yang, Yitao Liang +1
Low-Dimension-to-High-Dimension (LDHD) generalization is a special case of Out-of-Distribution (OOD) generalization, where the training data are restricted to a low-dimensional sub…
Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective
Yang Chen, Cong Fang, Zhouchen Lin +1
Foundation Models (FMs) have demonstrated remarkable insights into the relational dynamics of the world, leading to the crucial question: how do these models acquire an understandi…
DIGIC: Domain Generalizable Imitation Learning by Causal Discovery
Yang Chen, Yitao Liang, Zhouchen Lin
Causality has been combined with machine learning to produce robust representations for domain generalization. Most existing methods of this type require massive data from multiple…