Accelerating Constrained Decoding with Token Space Compression

Michael Sullivan, Alexander Koller

To guarantee that an LLM's outputs conform to a specified structure, context-free grammar (CFG) decoding engines force the selection of next tokens that produce strings that confor…