4 papers
Multi-Token Prediction and Alternative Tokenization Improve Learning Latent Multi-Hop Graph Search at Scale
Arvid Frydenlund, FuTe Wong, Katherine Lambert +1
Given an unseen graph, a start node, , and an end node, , can a language model generate the shortest path from to ? Prior work has found that models fail to learn this…
Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More
Arvid Frydenlund
This work concerns the path-star task, a minimal example of searching over a graph. The graph, , is star-shaped with arms radiating from a start node, . A language model…
The Mystery of the Pathological Path-star Task for Language Models
Arvid Frydenlund
The recently introduced path-star task is a minimal task designed to exemplify limitations to the abilities of language models (Bachmann and Nagarajan, 2024). It involves a path-st…
Language Modelling via Learning to Rank
Arvid Frydenlund, Gagandeep Singh, Frank Rudzicz
We consider language modelling (LM) as a multi-label structured prediction task by re-framing training from solely predicting a single ground-truth word to ranking a set of words w…