11 papers
Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining
Lukas Thede, Shengzhuang Chen, Stefan Winzeck +3
Continued pretraining enables language models to adapt to new domains and knowledge, but often at the cost of forgetting previously acquired capabilities. Replay can mitigate this…
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models
Lukas Thede, Yash Kumar Atri, David Chen +4
Constantly evolving real-world knowledge necessitates models to be updated continuously. Especially in medicine, as clinical evidence changes over time, outdated knowledge can pose…
Thomson: Continual Learning of Frontier Models for SovereignAI
Shengzhuang Chen, Jerrod Parker, Yejin Bang +23
The development of frontier models is commonly perceived to be the exclusive remit of a small number of heavily funded players, creating an information, economic and power asymmetr…
Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency
Yiran Huang, Lukas Thede, Massimiliano Mancini +2
While Large Vision Language Models (LVLMs) demonstrate impressive capabilities, their substantial computational and memory requirements pose deployment challenges on resource-const…
CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
Lukas Thede, Stefan Winzeck, Zeynep Akata +1
Large language model (LLM) post-training enhances latent skills, unlocks value alignment, improves performance, and enables domain adaptation. Unfortunately, post-training is known…
Attentive multilayer fusion for vision transformers
Laure Ciernik, Marco Morik, Lukas Thede +4
With the rise of large-scale foundation models, efficiently adapting them to downstream tasks remains a central challenge. Linear probing, which freezes the backbone and trains a l…
Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
Yiran Huang, Lukas Thede, Massimiliano Mancini +2
While Multimodal Large Language Models (MLLMs) demonstrate impressive capabilities, their substantial computational and memory requirements pose significant barriers to practical d…
WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
Lukas Thede, Karsten Roth, Matthias Bethge +2
Keeping large language models factually up-to-date is crucial for deployment, yet costly retraining remains a challenge. Knowledge editing offers a promising alternative, but metho…
Reflecting on the State of Rehearsal-free Continual Learning with Pretrained Models
Lukas Thede, Karsten Roth, Olivier J. Hénaff +2
With the advent and recent ubiquity of foundation models, continual learning (CL) has recently shifted from continual training from scratch to the continual adaptation of pretraine…
Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model
Karsten Roth, Lukas Thede, Almut Sophia Koepke +3
Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variati…
Learning to Defer with Limited Expert Predictions
Patrick Hemmer, Lukas Thede, Michael Vössing +2
Recent research suggests that combining AI models with a human expert can exceed the performance of either alone. The combination of their capabilities is often realized by learnin…