12 papers · 1 filter
InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
Fengze Liu, Weidong Zhou, Binbin Liu +7
Upweighting high-quality data in LLM pretraining often improves performance, but in datalimited regimes, especially under overtraining, stronger upweighting increases repetition an…
An Information-theoretic Propagation Denoising and Fusion Framework for Fake News Detection
Mengyang Chen, Lingwei Wei, Wei Zhou +1
Incomplete propagation data significantly hinders robust fake news detection. Recent approaches leverage large language models to simulate missing user interactions via role-playin…
Propagation Structure-Semantic Transfer Learning for Robust Fake News Detection
Mengyang Chen, Lingwei Wei, Han Cao +3
Fake news generally refers to false information that is spread deliberately to deceive people, which has detrimental social effects. Existing fake news detection methods primarily…
Diagnosing Retrieval Bias Under Multiple In-Context Knowledge Updates in Large Language Models
Boyu Qiao, Sean Guo, Xian Yang +4
LLMs are widely used in knowledge-intensive tasks where the same fact may be revised multiple times within context. Unlike prior work focusing on one-shot updates or single conflic…
MiLe Loss: a New Entropy-Weighed Loss for Mitigating the Bias of Learning Difficulties in Large Language Models
Zhenpeng Su, Xing Wu, Xue Bai +5
Generative language models are usually pretrained on large text corpus via predicting the next token (i.e., sub-word/word/phrase) given the previous ones. Recent works have demonst…
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
Minxuan Lv, Zhenpeng Su, Leiyu Pan +10
As large language models continue to scale, computational costs and resource consumption have emerged as significant challenges. While existing sparsification methods like pruning…