Regret Pre-training: Bridging Prior and Posterior Views for Enhanced Knowledge Grounding

Mingkuan Zhao, Xiayu Sun, Wentao Hu +5

Causal language models factorize sequence probabilities using only preceding context, leaving future information unexploited during training despite its availability in the trainin…