tokenize en · VERB
Pronunciation
- /ˈtoʊ.kən.aɪz/ (General-American)
Etymology
Etymology tree English token Proto-Indo-European *-id- Proto-Indo-European *-yéti Proto-Indo-European *-idyéti Proto-Hellenic *-íd͏̌d͏̌ō Ancient Greek -ῐ́ζω (-ĭ́zō)bor. Late Latin -izōder. Middle French -iserbor. Middle English -isen English -ize English tokenize From token + -ize.
Meanings
- (transitive) To reduce to a token or set of tokens by lexical analysis.
- (transitive) To substitute sensitive data with meaningless placeholders.
- (transitive) To treat as a token minority.
Forms
| Spelling | Features | Labels | Source |
|---|---|---|---|
| tokenise | alternative | lexicographic | |
| tokenizing | Tense=Pres, VerbForm=Part | lexicographic | |
| tokenized | Tense=Past, VerbForm=Part | lexicographic | |
| tokenized | Tense=Past | lexicographic | |
| tokenizes | Number=Sing, Person=3, Tense=Pres | lexicographic |
Coordinates
Deriveds
detokenize · retokenize · tokenizable · mistokenize · untokenized · tokenization · tokenizer