4 papers
Programs-of-Layers in LLMs through the Lens of Cortical Areas
Justus Westerhoff, Stephan Olbrich, Hatem Oraby +2
Inference in LLMs is conventionally a fixed-depth, fixed-order forward pass through every layer, regardless of how difficult the input is. The human brain does not work this way: u…
SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Models
Justus Westerhoff, Erblina Purelku, Jakob Hackstein +4
Typographic attacks exploit the interplay between text and visual content in multimodal foundation models, causing misclassifications when misleading text is embedded within images…
Robust Weight Imprinting: Insights from Neural Collapse and Proxy-Based Aggregation
Justus Westerhoff, Golzar Atefi, Mario Koddenbrock +4
The capacity of foundation models allows for their application to new, unseen tasks. The adaptation to such tasks is called transfer learning. An efficient transfer learning method…
Comply: Learning Sentences with Complex Weights inspired by Fruit Fly Olfaction
Alexei Figueroa, Justus Westerhoff, Golzar Atefi +5
Biologically inspired neural networks offer alternative avenues to model data distributions. FlyVec is a recent example that draws inspiration from the fruit fly's olfactory circui…