7 citations · 7 across the 4 of their papers we have counts for
4 papers
xHC: Expanded Hyper-Connections
Xiangdong Zhang, Xiaohan Qin, Sunan Zou +10
Hyper-Connections (HC) expand the residual stream of Transformers into parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained H…
NITP: Next Implicit Token Prediction for LLM Pre-training
Xiangdong Zhang, Debing Zhang, Shaofeng Zhang +3
Standard next-token prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse one-hot supervision leaves the l…
Towards Better Understanding of Disentangled Representations via Mutual Information
Xiaojiang Yang, Wendong Bi, Yitong Sun +2
Most existing works on disentangled representation learning are solely built upon an marginal independence assumption: all factors in disentangled representations should be statist…
INSET: Sentence Infilling with INter-SEntential Transformer
Yichen Huang, Yizhe Zhang, Oussama Elachqar +1
Missing sentence generation (or sentence infilling) fosters a wide range of applications in natural language generation, such as document auto-completion and meeting note expansion…