19 citations · 74 across the 10 of their papers we have counts for
10 papers
SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models
Vithursan Thangarasa, Abhay Gupta, William Marshall +5
The pre-training and fine-tuning paradigm has contributed to a number of breakthroughs in Natural Language Processing (NLP). Instead of directly training on a downstream task, lang…
Do we need Label Regularization to Fine-tune Pre-trained Language Models?
Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh +5
Knowledge Distillation (KD) is a prominent neural model compression technique that heavily relies on teacher network predictions to guide the training of a student model. Consideri…
A Short Study on Compressing Decoder-Based Language Models
Tianda Li, Yassir El Mesbahi, Ivan Kobyzev +6
Pre-trained Language Models (PLMs) have been successful for a wide range of natural language processing (NLP) tasks. The state-of-the-art of PLMs, however, are extremely large to b…
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Tianda Li, Ahmad Rashid, Aref Jafari +3
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one. Even though KD has shown promise on a w…
Unsupervised Pre-training with Structured Knowledge for Improving Natural Language Inference
Xiaoyu Yang, Xiaodan Zhu, Zhan Shi +1
While recent research on natural language inference has considerably benefited from large annotated datasets, the amount of inference-related knowledge (including commonsense) prov…
Learning to Retrieve Entity-Aware Knowledge and Generate Responses with Copy Mechanism for Task-Oriented Dialogue Systems
Chao-Hong Tan, Xiaoyu Yang, Zi'ou Zheng +7
Task-oriented conversational modeling with unstructured knowledge access, as track 1 of the 9th Dialogue System Technology Challenges (DSTC 9), requests to build a system to genera…