85 citations · 273 across the 13 of their papers we have counts for
4 papers · 1 filter
Lasting Effects of Abstract Pretraining Beyond Perplexity
Zachary Shinnick, Hemanth Saratchandran, Damien Teney +1
Language models are typically pretrained from random initialization. Recent work challenges this convention, showing that a brief warm-up on abstract, algorithmically generated dat…
Procedural Core: A Compact Recurrent Initialization for Vision Transformers
Zachary Shinnick, Christian Internò, Hemanth Saratchandran +2
Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work showed that a small amount of a…
Back from the Future: Key-Value Cache Management by Counter-Causal Surprise
Stephen Gould, Anton van den Hengel
Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years. Computational demands of large language…
Deep Weakly-supervised Anomaly Detection
Guansong Pang, Chunhua Shen, Huidong Jin +1
Recent semi-supervised anomaly detection methods that are trained using small labeled anomaly examples and large unlabeled data (mostly normal data) have shown largely improved per…