Showing 2024Show all
2 papers · 1 filter
cs.CL2024
Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Seungwook Han, Jinyeop Song, Jeff Gore +1
Autoregressive transformers exhibit adaptive learning through in-context learning (ICL), which begs the question of how. Prior works have shown that transformers represent the ICL…
cs.LG2024
A Resource Model For Neural Scaling Law
Jinyeop Song, Ziming Liu, Max Tegmark +1
Neural scaling laws characterize how model performance improves as the model size scales up. Inspired by empirical observations, we introduce a resource model of neural scaling. A…