Skip to main content

tokenize_chars

use_token

Function tokenize_chars 

Source 
pub fn tokenize_chars(input: &str) -> Vec<Token>
Expand description

Splits input into Unicode scalar values.