Pointer-Guided Pre-Training: Infusing Large Language Models with Paragraph-Level Contextual Awareness

Lars Hillebrand, Prabhupad Pradhan, Christian Bauckhage +1

We introduce "pointer-guided segment ordering" (SO), a novel pre-training technique aimed at enhancing the contextual understanding of paragraph-level text representations in large…