134 citations · 189 across the 22 of their papers we have counts for
18 papers · 1 filter
AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
Ziqiang Cui, Han Shi, Bowei He +8
Multi-Token Prediction (MTP) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads, training the model to predict several f…
Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
Yusheng Zhao, Hourun Li, Bohan Wu +5
The attention mechanism has been the core component in modern transformer architectures. However, the computation of standard full attention scales quadratically with the sequence…
Safe: Enhancing Mathematical Reasoning in Large Language Models via Retrospective Step-aware Formal Verification
Chengwu Liu, Ye Yuan, Yichun Yin +7
Chain-of-Thought (CoT) prompting has become the de facto method to elicit reasoning capabilities from large language models (LLMs). However, to mitigate hallucinations in CoT that…
TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models
Jing Xiong, Jianhao Shen, Ye Yuan +11
Automated theorem proving (ATP) has become an appealing domain for exploring the reasoning ability of the recent successful generative language models. However, current ATP benchma…
AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models
Siheng Li, Cheng Yang, Yichun Yin +6
Information-seeking conversation, which aims to help users gather information through conversation, has achieved great progress in recent years. However, the research is still stym…
NewsDialogues: Towards Proactive News Grounded Conversation
Siheng Li, Yichun Yin, Cheng Yang +7
Hot news is one of the most popular topics in daily conversations. However, news grounded conversation has long been stymied by the lack of well-designed task definition and scarce…
G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks
Zhongwei Wan, Yichun Yin, Wei Zhang +5
Recently, domain-specific PLMs have been proposed to boost the task performance of specific domains (e.g., biomedical and computer science) by continuing to pre-train general PLMs…
FPT: Improving Prompt Tuning Efficiency via Progressive Training
Yufei Huang, Yujia Qin, Huadong Wang +4
Recently, prompt tuning (PT) has gained increasing attention as a parameter-efficient way of tuning pre-trained language models (PLMs). Despite extensively reducing the number of t…
bert2BERT: Towards Reusable Pretrained Language Models
Cheng Chen, Yichun Yin, Lifeng Shang +7
In recent years, researchers tend to pre-train ever-larger language models to explore the upper limit of deep models. However, large language model pre-training costs intensive com…
Generate & Rank: A Multi-task Framework for Math Word Problems
Jianhao Shen, Yichun Yin, Lin Li +4
Math word problem (MWP) is a challenging and critical task in natural language processing. Many recent studies formalize MWP as a generation task and have adopted sequence-to-seque…
Integrating Regular Expressions with Neural Networks via DFA
Shaobo Li, Qun Liu, Xin Jiang +5
Human-designed rules are widely used to build industry applications. However, it is infeasible to maintain thousands of such hand-crafted rules. So it is very important to integrat…
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Yichun Yin, Cheng Chen, Lifeng Shang +3
Pre-trained language models (PLMs) have achieved great success in natural language processing. Most of PLMs follow the default setting of architecture hyper-parameters (e.g., the h…
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
Cheng Chen, Yichun Yin, Lifeng Shang +4
Task-agnostic knowledge distillation, a teacher-student framework, has been proved effective for BERT compression. Although achieving promising results on NLP tasks, it requires en…
LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
Xiaoqi Jiao, Yichun Yin, Lifeng Shang +5
The multilingual pre-trained language models (e.g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks. However, these mod…
Improving Task-Agnostic BERT Distillation with Layer Mapping Search
Xiaoqi Jiao, Huating Chang, Yichun Yin +6
Knowledge distillation (KD) which transfers the knowledge from a large teacher model to a small student model, has been widely used to compress the BERT model recently. Besides the…
TernaryBERT: Distillation-aware Ultra-low Bit BERT
Wei Zhang, Lu Hou, Yichun Yin +4
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks.However, these models are both computation and memory…
PoD: Positional Dependency-Based Word Embedding for Aspect Term Extraction
Yichun Yin, Chenguang Wang, Ming Zhang
Dependency context-based word embedding jointly learns the representations of word and dependency context, and has been proved effective in aspect term extraction. In this paper, w…
TinyBERT: Distilling BERT for Natural Language Understanding
Xiaoqi Jiao, Yichun Yin, Lifeng Shang +5
Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually c…