Conceptizer: Next-Token Prediction Is Not All You Need

Haoran Zheng, Chen Shani

Next-token prediction (NTP) rewards only the token that occurs next, penalizing every alternative. But natural language often offers many ways to express the same meaning. We intro…