Unlocking Full Efficiency of Token Filtering in Large Language Model Training

Di Chai, Pengbo Li, Feiyuan Zhang +7

Token filtering has been proposed to enhance the utility of large language models (LLMs) by eliminating inconsequential tokens during training. While usingfewer tokens is expected…