Flow of Spans: Generalizing Language Models to Dynamic Span-Vocabulary via GFlowNets

Bo Xue, Yunchong Song, Fanghao Shao +5

Standard autoregressive language models generate text token-by-token from a fixed vocabulary, inducing a tree-structured state space when viewing token sampling as an action, which…