1.1k citations · 1.2k across the 35 of their papers we have counts for
16 papers · 1 filter
Fast Quiet-STaR: Thinking Without Thought Tokens
Wei Huang, Yizhe Xiong, Xin Ye +4
Large Language Models (LLMs) have achieved impressive performance across a range of natural language processing tasks. However, recent advances demonstrate that further gains parti…
Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Experts
Leiyu Pan, Zhenpeng Su, Minxuan Lv +10
Large language models have demonstrated exceptional performance across a wide range of tasks. However, dense models usually suffer from sparse activation, where many activation val…
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
Minxuan Lv, Zhenpeng Su, Leiyu Pan +10
As large language models continue to scale, computational costs and resource consumption have emerged as significant challenges. While existing sparsification methods like pruning…
UniAttn: Reducing Inference Costs via Softmax Unification for Post-Training LLMs
Yizhe Xiong, Wei Huang, Xin Ye +6
Post-training is essential for adapting Large Language Models (LLMs) to real-world applications. Deploying post-trained models faces significant challenges due to substantial memor…
LBPE: Long-token-first Tokenization to Improve Large Language Models
Haoran Lian, Yizhe Xiong, Zijia Lin +5
The prevalent use of Byte Pair Encoding (BPE) in Large Language Models (LLMs) facilitates robust handling of subword units and avoids issues of out-of-vocabulary words. Despite its…
MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Experts
Zhenpeng Su, Zijia Lin, Xue Bai +8
Scaling the size of a model enhances its capabilities but significantly increases computation complexity. Mixture-of-Experts models (MoE) address the issue by allowing model size t…
Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
Haoran Lian, Yizhe Xiong, Jianwei Niu +7
Byte Pair Encoding (BPE) serves as a foundation method for text tokenization in the Natural Language Processing (NLP) field. Despite its wide adoption, the original BPE algorithm h…
Temporal Scaling Law for Large Language Models
Yizhe Xiong, Xiansheng Chen, Xin Ye +8
Recently, Large Language Models (LLMs) have been widely adopted in a wide range of tasks, leading to increasing attention towards the research on how scaling LLMs affects their per…
MiLe Loss: a New Entropy-Weighed Loss for Mitigating the Bias of Learning Difficulties in Large Language Models
Zhenpeng Su, Xing Wu, Xue Bai +5
Generative language models are usually pretrained on large text corpus via predicting the next token (i.e., sub-word/word/phrase) given the previous ones. Recent works have demonst…
CoT-MAE v2: Contextual Masked Auto-Encoder with Multi-view Modeling for Passage Retrieval
Xing Wu, Guangyuan Ma, Peng Wang +4
Growing techniques have been emerging to improve the performance of passage retrieval. As an effective representation bottleneck pretraining technique, the contextual masked auto-e…
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings
Xing Wu, Chaochen Gao, Zijia Lin +3
Contrastive learning has been extensively studied in sentence embedding learning, which assumes that the embeddings of different views of the same sentence are closer. The constrai…
ConTextual Masked Auto-Encoder for Dense Passage Retrieval
Xing Wu, Guangyuan Ma, Meng Lin +3
Dense passage retrieval aims to retrieve the relevant passages of a query from a large corpus based on dense representations (i.e., vectors) of the query and the passages. Recent s…
UniTrans: Unifying Model Transfer and Data Transfer for Cross-Lingual Named Entity Recognition with Unlabeled Data
Qianhui Wu, Zijia Lin, Börje F. Karlsson +2
Prior works in cross-lingual named entity recognition (NER) with no/little labeled data fall into two primary categories: model transfer based and data transfer based methods. In t…
Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language
Qianhui Wu, Zijia Lin, Börje F. Karlsson +2
To better tackle the named entity recognition (NER) problem on languages with little/no labeled data, cross-lingual NER must effectively leverage knowledge learned from source lang…
Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resources
Qianhui Wu, Zijia Lin, Guoxin Wang +4
For languages with no annotated resources, transferring knowledge from rich-resource languages is an effective solution for named entity recognition (NER). While all existing metho…
GRN: Gated Relation Network to Enhance Convolutional Neural Network for Named Entity Recognition
Hui Chen, Zijia Lin, Guiguang Ding +3
The dominant approaches for named entity recognition (NER) mostly adopt complex recurrent neural networks (RNN), e.g., long-short-term-memory (LSTM). However, RNNs are limited by t…