9 citations · 20 across the 20 of their papers we have counts for
9 papers · 1 filter
Virtual Width Networks
Seed, Baisheng Li, Banggu Wu +115
We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without incurring the quadratic cost of increasing the hidden size. VWN d…
UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning
Zihao Huang, Yu Bao, Qiyang Min +8
While Mixture of Experts (MoE) models achieve remarkable efficiency by activating only subsets of parameters, they suffer from high memory access costs during inference. Memory-lay…
Scaling Law for Quantization-Aware Training
Mengzhao Chen, Chaoyi Zhang, Jing Liu +8
Large language models (LLMs) demand substantial computational and memory resources, creating deployment challenges. Quantization-aware training (QAT) addresses these challenges by…
Stepsize anything: A unified learning rate schedule for budgeted-iteration training
Anda Tang, Yiming Dong, Yutao Zeng +2
The expanding computational costs and limited resources underscore the critical need for budgeted-iteration training, which aims to achieve optimal learning within predetermined it…
Frac-Connections: Fractional Extension of Hyper-Connections
Defa Zhu, Hongzhi Huang, Jundong Zhou +5
Residual connections are central to modern deep learning architectures, enabling the training of very deep networks by mitigating gradient vanishing. Hyper-Connections recently gen…
Ultra-Sparse Memory Network
Zihao Huang, Qiyang Min, Hongzhi Huang +4
It is widely acknowledged that the performance of Transformer models is logarithmically related to their number of parameters and computational complexity. While approaches like Mi…
Hyper-Connections
Defa Zhu, Hongzhi Huang, Zihao Huang +5
We present hyper-connections, a simple yet effective method that can serve as an alternative to residual connections. This approach specifically addresses common drawbacks observed…
KnowCoder: Coding Structured Knowledge into LLMs for Universal Information Extraction
Zixuan Li, Yutao Zeng, Yuxin Zuo +14
In this paper, we propose KnowCoder, a Large Language Model (LLM) to conduct Universal Information Extraction (UIE) via code generation. KnowCoder aims to develop a kind of unified…
What is Event Knowledge Graph: A Survey
Saiping Guan, Xueqi Cheng, Long Bai +5
Besides entity-centric knowledge, usually organized as Knowledge Graph (KG), events are also an essential kind of knowledge in the world, which trigger the spring up of event-centr…