N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs

Ilya Zisman, Alexander Nikulin, Viacheslav Sinii +5

In-context learning allows models like transformers to adapt to new tasks from a few examples without updating their weights, a desirable trait for reinforcement learning (RL). How…