IRS for Computer Character Sequences Filtration: a new software tool and algorithm to support the IRS at tokenization process

Ahmad Al Badawi, Qasem Abu Al-Haija

Tokenization is the task of chopping it up into pieces, called tokens, perhaps at the same time throwing away certain characters, such as punctuation. A token is an instance of tok…