4 papers
Can Machines Really See Objects in Images? A Study Based on Syntactic Distance and Visual Self-Referential Instances
Xingyu Peng, Junran Wu, Yue Hou +9
Can a vision model truly see an object, or does it only fit surface-level visual cues? Following Wittgenstein's view that the limits of language are the limits of the world, we vie…
MiniLLM: On-Policy Distillation of Large Language Models
Yuxian Gu, Li Dong, Furu Wei +1
Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied…
Differential Transformer
Tianzhu Ye, Li Dong, Yuqing Xia +4
Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling no…
Direct Preference Knowledge Distillation for Large Language Models
Yixing Li, Yuxian Gu, Li Dong +3
In the field of large language models (LLMs), Knowledge Distillation (KD) is a critical technique for transferring capabilities from teacher models to student models. However, exis…