8 papers
Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Jie Wang, Shiwei Luo, Qi Zhang +1
Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computati…
DVAGen: Dynamic Vocabulary Augmented Generation
Wei Du, Nuowei Liu, Jie Wang +4
Language models trained with a fixed vocabulary struggle to generalize to novel or out-of-vocabulary words, limiting their flexibility in handling diverse token combinations. Exist…
TASO: Task-Aligned Sparse Optimization for Parameter-Efficient Model Adaptation
Daiye Miao, Yufang Liu, Jie Wang +6
LoRA has become one of the most widely used parameter-efficient fine-tuning methods due to its simplicity and effectiveness. However, numerous studies have shown that LoRA often in…
PDFBench: A Benchmark for De novo Protein Design from Function
Jiahao Kuang, Nuowei Liu, Jie Wang +3
Function-guided protein design is a crucial task with significant applications in drug discovery and enzyme engineering. However, the field lacks a unified and comprehensive evalua…
Federated Neural Architecture Search with Model-Agnostic Meta Learning
Xinyuan Huang, Jiechao Gao
Federated Learning (FL) often struggles with data heterogeneity due to the naturally uneven distribution of user data across devices. Federated Neural Architecture Search (NAS) ena…
AntLM: Bridging Causal and Masked Language Models
Xinru Yu, Bin Guo, Shiwei Luo +3
Causal Language Modeling (CLM) and Masked Language Modeling (MLM) are two mainstream learning paradigms based on Transformer networks, specifically the Decoder-only and Encoder-onl…
H-FedSN: Personalized Sparse Networks for Efficient and Accurate Hierarchical Federated Learning for IoT Applications
Jiechao Gao, Yuangang Li, Jie Wang +3
With the rapid development of the Internet of Things (IoT), federated learning (FL) has gained increasing attention for its privacy-preserving use of distributed data. However, con…
Length Generalization of Causal Transformers without Position Encoding
Jie Wang, Tao Ji, Yuanbin Wu +5
Generalizing to longer sentences is important for recent Transformer-based language models. Besides algorithms manipulating explicit position features, the success of Transformers…