CatGPT2 is a 1.2 million parameter Large Language Model running entirely in Scratch. It's a completion model, not a chatbot. Give it a prompt like "once upon a time" and it will continue writing a (semi-coherent) story from there. Chat-style models are difficult at this small scale (they tend to just memorize topics and responses, breaking down on anything new), so CatGPT2 focuses on story writing instead, where it performs much better. This is a sequel to my original LLM project, CatGPT (https://scratch.mit.edu/projects/1288884594/). CatGPT2 is over 100x larger, but runs nearly as fast thanks to optimizations in my cattorch library and other changes.
Trained in PyTorch on the roneneldan/TinyStories dataset. Exported to Scratch using my own PyTorch-to-Scratch transpiler, cattorch (https://pypi.org/project/cattorch/) Text rendered using "Stamp Font Engine" by -Rex- (https://scratch.mit.edu/projects/349634648/)
