activity
20232026
most citedHow Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations

2 citations · 2 across the 12 of their papers we have counts for

collaborators
Showing 2026Show all

2 papers · 1 filter

cs.CL2026

Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention

Xiao Li, Chengruidong Zhang, Hao Luo +15

Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active corr…

cs.LG2026

Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws

Zhiwei Xu, Shihao Wu, Hanseul Cho +2

Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant data and a single pass over t…