This is a improved Unigram tokeniser. I did make a few smal changes to the base algorithm. Like pre tokenising the text into words. I also score tokens based on the compression gains instead of raw frequency.
Pre trained dataset: |_|A|B|C|D|E|F|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U|V|W|X|Y|Z|.|!|?|'|OU|_Y|_I|_T|_A|_W|_TO|RE|HA|_M|_D|HAT|_S|_I'|IN|_DO|VE|_I'M|_ME|ING|_L|OW|_TH|_O|AN|_IS|_B|EE|OUR|_G|_SO|_F|_THE|_C|_N|AY|_GO|OU'|KE|LL|_LO|IT|AT|'S|_BE|_OF|_AN|_WH|_MY|_AM|LD|_H|_HA|_ME?|_TOD|_LI|INK|OT|EEL|OR|'T|_E|_WHO|_P|ITH|_ANY|ME|OU?|OUT|_AB|VER|_K|_U|_J|NE|_SH|_IN|_WE|ER|OD|_KN|_PL|AS|AY?|IS|_IT|TH|ANT|ORE|_DON|AL|TT|AR|ST|LY|_JO|AI|_AS|_TE|_OR|OK|EL|_MYS|ELF|_ON|_BA|VERY|_US|_OFF|_OU|_OUT|_WI|_R|GH|_RI|GHT|ET|EED|_JU|EN|_AR|_ARE|EEM|PP|PPY|TTY|_?|_IF|MEO|_!|A.|CE|FI|KD|LT|QN|R.|S.|OWN|_CH|ITC|ORI|TE|_ED|A.P|R.S|_MA|_ST|OP|CH|_WHI|_WHY|_ASK|ND|NN|_FU|NNY|_RE|_UP|_ANYO|_TA|ANN|LK|_I'D|_TI|FE|_PLE|ASE|_AREN|SW|CK|_AT|_ONE|EEN|TW|_UG|AY.|AV|_NO|_NOT|_NOW|EW|ID|EK|_ANYB|ODY|_BAD|_AND|GHT?|_AV|AB|LE|ON|AIL|_KI|WA|KE!|OU'RE|OU'R|_GOING|_GOOD|_GOO|_GOIN|_GOI|LLY|_REA|_REAL|_ANYON|_ANYONE|_TAK|_TAKE|_TALK|_TAL|_TEL|_TELL|OKE|AIR.S|AIR.|AIR|_JOKE|_JOK|NEE|NEW|_SHO|_SHOU This was trained on the using the “chit-chat-dataset” you can find the dataset at: “https://github.com/microsoft/botframework-cli/blob/main/packages/qnamaker/docs/chit-chat-dataset.md” I scored it using the eval dataset at the bottom. Of the credits. It scored 1,76 on the same dataset the GPT-4 tokeniser scored: 5.45 Eval dataset: the quick brown fox jumps over the lazy dog. hello world! how are you doing today? I am learning about tokenization and NLP. tokens are the building blocks of natural language. machine learning is exciting, and so is deep learning. Python is a powerful language for AI development. let's create a tokenizer that splits words efficiently. are we ready to start the training process? this is a test sentence for tokenizing words. tokenize this sentence into individual components! tokenization helps in breaking down text into smaller pieces. subwords can be helpful when dealing with rare words. I wonder if this tokenizer will work well on longer texts. efficient tokenization is crucial for machine learning models. the sun rises in the east, and sets in the west. could you pass me the salt, please? this model is designed to handle multiple languages. I love reading books, especially science fiction. tokenization works better when it's context-aware. you must understand tokenization to build NLP models. in this project, we are focusing on improving the tokenizer. have you considered the performance of your tokenizer yet? tokenizers are essential for text preprocessing in NLP. don't forget to optimize your tokenizer for speed and accuracy. let's break down this long sentence into meaningful tokens. tokenization can be tricky with compound words. learning about subwords is crucial when building tokenizers. many languages have different structures that affect tokenization. I enjoy experimenting with different machine learning models. this text is a sample input for the tokenizer to process. whether you are working on sentiment analysis or translation, tokenization is key. let's use this tokenizer to process a few more sentences. tokenize sentences like "I am going to the store" efficiently. now, let's test the tokenizer with some punctuation. how do you deal with numbers in tokenization? special characters can sometimes create issues in tokenization. there are many strategies for handling punctuation in NLP. subword tokenization methods like BPE are useful in many cases. the more diverse the dataset, the better the tokenizer will perform. text segmentation is closely related to tokenization. try experimenting with different tokenization strategies to find what works best. handling compound words and contractions is an interesting challenge. I wonder how the tokenizer handles ambiguous words in different contexts. it's important to test tokenizers on real-world data for better performance. tokenizers can be used in various applications, including text classification. this tokenizer should be flexible enough to handle various text types. sentence tokenization is a different challenge than word tokenization. have you tried using a pre-trained tokenizer for comparison? some tokenizers are designed to work specifically with particular languages. if we optimize our tokenizer, it should work well on longer sentences. let's explore how tokenization is used in speech recognition. tokenizers can also be useful for information retrieval tasks. we will apply our tokenizer to process large datasets soon. the tokenizer should be able to handle edge cases like typos and unusual words. text normalization is an important preprocessing step before tokenization. there are different ways to handle tokenization based on the task. it’s fascinating how simple tokenization can help in building powerful NLP systems.
