4 papers · 1 filter
Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining
Rupert Mitchell, Kristian Kersting
Pretraining transformers on long sequences (entire code repositories, collections of related documents) is bottlenecked by quadratic attention costs. We present Multipole Semantic…
Where is the Truth? The Risk of Getting Confounded in a Continual World
Florian Peter Busch, Roshni Kamath, Rupert Mitchell +3
A dataset is confounded if it is most easily solved via a spurious correlation, which fails to generalize to new data. In this work, we show that, in a continual learning setting w…
BOWL: A Deceptively Simple Open World Learner
Roshni . R. Kamath, Rupert Mitchell, Subarnaduti Paul +2
Traditional machine learning excels on static benchmarks, but the real world is dynamic and seldom as carefully curated as test sets. Practical applications may generally encounter…
Continual Learning Should Move Beyond Incremental Classification
Rupert Mitchell, Antonio Alliegro, Raffaello Camoriano +17
Continual learning (CL) is the sub-field of machine learning concerned with accumulating knowledge in dynamic environments. So far, CL research has mainly focused on incremental cl…