StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training

Kaustubh Ponkshe, Venkatapathy Subramanian, Natwar Modani +1

Most state-of-the-art techniques for Language Models (LMs) today rely on transformer-based architectures and their ubiquitous attention mechanism. However, the exponential growth i…