Skip to main content

tokenize_words

use_token

Function tokenize_words 

Source 
pub fn tokenize_words(input: &str) -> Vec<Token>
Expand description

Extracts conservative word tokens.