tokenizer
About 0.01 uses per million words.
Meaning
- noun A system that parses an input stream into its component tokens.
Similar words
tokeniser parser lexical analysis terp serializer bracketer detokenizer unparser parse lexical analyzer
Often followed by
to is and for that
Save “tokenizer” to learn later