Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks

Abstract

Curriculum Learning emphasizes the order of training instances in a computational learning setup. The core hypothesis is that simpler instances should be learned early as building blocks to learn more complex ones. Despite its usefulness, it is still unknown how exactly the internal representation of models are affected by curriculum learning. In this paper, we study the effect of curriculum learning on Long Short-Term Memory (LSTM) networks, which have shown strong competency in many Natural Language Processing (NLP) problems. Our experiments on sentiment analysis task and a synthetic task similar to sequence prediction tasks in NLP show that curriculum learning has a positive effect on the LSTM's internal states by biasing the model towards building constructive representations i.e. the internal representation at the previous timesteps are used as building blocks for the final prediction. We also find that smaller models significantly improves when they are trained with curriculum learning. Lastly, we show that curriculum learning helps more when the amount of training data is limited.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Convolutional Neural Networks for Sentence Classification
• Visualizing and Understanding Recurrent Networks
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory
• Grid Long Short-Term Memory

Cited by in corpus (11)

• An Empirical Exploration of Curriculum Learning for Neural Machine Translation
• Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
• A Survey on Curriculum Learning
• When Do Curricula Work?
• Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent
• Understanding and Controlling Memory in Recurrent Neural Networks
• Self-paced Data Augmentation for Training Neural Networks
• An Empirical Comparison of Syllabuses for Curriculum Learning
• Self-Guided Curriculum Learning for Neural Machine Translation
• Data Ordering Patterns for Neural Machine Translation: An Empirical Study
• Analyzing Curriculum Learning for Sentiment Analysis along Task Difficulty, Pacing and Visualization Axes