A Survey on Symbolic Knowledge Distillation of Large Language Models

Abstract

This survey paper delves into the emerging and critical area of symbolic knowledge distillation in Large Language Models (LLMs). As LLMs like Generative Pre-trained Transformer-3 (GPT-3) and Bidirectional Encoder Representations from Transformers (BERT) continue to expand in scale and complexity, the challenge of effectively harnessing their extensive knowledge becomes paramount. This survey concentrates on the process of distilling the intricate, often implicit knowledge contained within these models into a more symbolic, explicit form. This transformation is crucial for enhancing the interpretability, efficiency, and applicability of LLMs. We categorize the existing research based on methodologies and applications, focusing on how symbolic knowledge distillation can be used to improve the transparency and functionality of smaller, more efficient Artificial Intelligence (AI) models. The survey discusses the core challenges, including maintaining the depth of knowledge in a comprehensible format, and explores the various approaches and techniques that have been developed in this field. We identify gaps in current research and potential opportunities for future advancements. This survey aims to provide a comprehensive overview of symbolic knowledge distillation in LLMs, spotlighting its significance in the progression towards more accessible and efficient AI systems.

21 pages, 7 figures

References in corpus (81)

• Efficient Estimation of Word Representations in Vector Space
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• Knowledge Distillation: A Survey
• PaLM: Scaling Language Modeling with Pathways
• FitNets: Hints for Thin Deep Nets
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• A Survey of Large Language Models
• Scaling Laws for Neural Language Models
• Evaluating Large Language Models Trained on Code
• Universal Sentence Encoder
• Scaling Instruction-Finetuned Language Models
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• LaMDA: Language Models for Dialog Applications
• Training Compute-Optimal Large Language Models
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Attention is not Explanation
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• GLM-130B: An Open Bilingual Pre-trained Model
• Release Strategies and the Social Impacts of Language Models
• Solving Quantitative Reasoning Problems with Language Models
• Galactica: A Large Language Model for Science
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• A Survey on Evaluation of Large Language Models
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Locating and Editing Factual Associations in GPT
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
• Improving alignment of dialogue agents via targeted human judgements
• LIMA: Less Is More for Alignment
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• WizardLM: Empowering large pre-trained language models to follow complex instructions
• WT5?! Training Text-to-Text Models to Explain their Predictions
• Instruction Tuning for Large Language Models: A Survey
• UL2: Unifying Language Learning Paradigms
• Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
• Will we run out of data? Limits of LLM scaling based on human-generated data
• Ask Me Anything: A simple strategy for prompting language models
• Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4
• A Review on Language Models as Knowledge Bases
• Neurosymbolic Reinforcement Learning and Planning: A Survey
• Aligning Large Language Models with Human: A Survey
• Specializing Smaller Language Models towards Multi-Step Reasoning
• Knowledge Neurons in Pretrained Transformers
• Modifying Memories in Transformer Models
• Alignment of Language Agents
• Lawyer LLaMA Technical Report
• Towards Continual Knowledge Learning of Language Models
• Large Language Models for Robotics: A Survey
• Graph-based Knowledge Distillation by Multi-head Attention Network
• A Survey on Model Compression for Large Language Models
• ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Large Language Models for Software Engineering: Survey and Open Problems
• DARWIN Series: Domain Specific Large Language Models for Natural Science
• Orca 2: Teaching Small Language Models How to Reason
• WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
• Measuring Systematic Generalization in Neural Proof Generation with Transformers
• A Survey on Verification and Validation, Testing and Evaluations of Neurosymbolic Artificial Intelligence
• Formal Mathematics Statement Curriculum Learning
• Scaling Laws and Interpretability of Learning from Repeated Data
• Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation
• Language Models in the Loop: Incorporating Prompting into Weak Supervision
• Reinforced Self-Training (ReST) for Language Modeling
• Language Models As or For Knowledge Bases
• Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
• Self-Rewarding Language Models
• QAmeleon: Multilingual QA with Only 5 Examples
• Multimodal Large Language Models: A Survey
• Lion: Adversarial Distillation of Proprietary Large Language Models
• PB-LLM: Partially Binarized Large Language Models
• RuleBert: Teaching Soft Rules to Pre-trained Language Models
• SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models
• Localized Symbolic Knowledge Distillation for Visual Commonsense Models

Cited by in corpus (1)

• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions