Publications (74)
Unshackling Context Length: An Efficient Selective Attention Approach through Query-Key Compression
Haoyu Wang, Tong Teng, Tianyu Guo +4
Handling long-context sequences efficiently remains a significant challenge in large language models (LLMs). Existing methods for token selection in sequence extrapolation either e…
PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference
Xutao Wang, Hanting Chen, Tianyu Guo +1
The paper proposes PUe, a framework that improves positive‑unlabeled (PU) learning under biased label selection by using normalized propensity scores and inverse probability weight…
Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
Miao Rang, Zhenni Bi, Hang Zhou +6
Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, t…
AdderNet: Do We Really Need Multiplications in Deep Learning?
Hanting Chen, Yunhe Wang, Chunjing Xu +4
Compared with cheap addition operation, multiplication operation is of much higher computation complexity. The widely-used convolutions in deep neural networks are exactly cross-co…
Data Upcycling Knowledge Distillation for Image Super-Resolution
Yun Zhang, Wei Li, Simiao Li +6
Knowledge distillation (KD) compresses deep neural networks by transferring task-related knowledge from cumbersome pre-trained teacher models to compact student models. However, cu…
Multi-Granularity Semantic Revision for Large Language Model Distillation
Xiaoyu Liu, Yun Zhang, Wei Li +7
Knowledge distillation plays a key role in compressing the Large Language Models (LLMs), which boosts a small-size student model under large teacher models' guidance. However, exis…