Transformer Is Inherently a Causal Learner

Xinyue Wang, Stephen Wang, Biwei Huang

We reveal that transformers trained in an autoregressive manner naturally encode time-delayed causal structures in their learned representations. When predicting future values in m…