4 papers
Task-Level Natural Language Priors as Learning Signals for Low-Resource LLM Training
Jian Gao, Xiao Zhang, Xun Zhu +2
Large language models (LLMs) often struggle when low-resource training data are ambiguous or incomplete. Task-level natural-language priors can provide useful guidance in such sett…
Enhancing elusive clues in knowledge learning by contrasting attention of language models
Jian Gao, Xiao Zhang, Ji Wu +1
Causal language models acquire vast amount of knowledge from general text corpus during pretraining, but the efficiency of knowledge learning is known to be unsatisfactory, especia…
Co-occurrence is not Factual Association in Language Models
Xiao Zhang, Miao Li, Ji Wu
Pretrained language models can encode a large amount of knowledge and utilize it for various reasoning tasks, yet they can still struggle to learn novel factual knowledge effective…
Conditional Language Learning with Context
Xiao Zhang, Miao Li, Ji Wu
Language models can learn sophisticated language understanding skills from fitting raw text. They also unselectively learn useless corpus statistics and biases, especially during f…