DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integration

Martin Kuo, Jianyi Zhang, Dongting Li +1

Pretraining language models is still a challenge for many researchers due to its substantial computational costs. As such, there is growing interest in developing more affordable p…