This is a BPE tokenizer. it takes a prompt then find where each letter pair of your prompt is in the list of words. you can add words to the list, and type any prompt. if you use, give credit! future of my ai i will have tokens with embeds
try "let's go to the store" if it doesn't have a word in its list it will use letters. if it doesn't have a letter in the list it will break
