12 citations · 22 across the 3 of their papers we have counts for
3 papers
Continual Pre-Training of Large Language Models: How to (re)warm your model?
Kshitij Gupta, Benjamin Thérien, Adam Ibrahim +5
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more effici…
Beyond Supervised Continual Learning: a Review
Benedikt Bagus, Alexander Gepperth, Timothée Lesort
Continual Learning (CL, sometimes also termed incremental learning) is a flavor of machine learning where the usual assumption of stationary data distribution is relaxed or omitted…
Continual Learning with Foundation Models: An Empirical Study of Latent Replay
Oleksiy Ostapenko, Timothee Lesort, Pau Rodríguez +4
Rapid development of large-scale pre-training has resulted in foundation models that can act as effective feature extractors on a variety of downstream tasks and domains. Motivated…