AxiaLM 1.1 | "The Future of Scratch AI." [ use turbowarp; https://turbowarp.org/#1392418938 ] ================================ > AxiaLM 1.1 is my biggest and most powerful AI ever, boasting 1.7 million parameters > As of 1.1, AxiaLM models are now conversational! You can now have a chat with them instead of them only predicting text. > While it's a massive breakthrough compared to older AxiaLM models, it's still tiny! Don't expect this to talk like ChatGPT. ================================ Specifications: > 1,689,216 Parameters > Decoder Transformer > 2,048 BPE Token Vocabulary > 128 Hidden Dimension > 448 Hidden Size > 6 Attention Layers > 4 Attention Heads > 512 Token Context Length > Trained on 40M Tokens of Data > 4-bit Quantization Merged with Custom Compression for Weights, Scales, and Norm Storing. ================================ Examples you *can* ask it: > "Hi" > "Who are you?" > "Solve 4 plus 8" > "What can you do?" Examples you *cannot* ask it: > "Explain quantum physics" > "What is a refrigerator?" > "What day is it?" > [Any coding] > "Solve for X" ================================ Notes: > This thumbnail promised more, I know! But 1.1 isn't the model that I plan to represent the "future of scratch AI", but rather the core first steps. You'll start to notice huge improvement when I release 1.2 PRO, which has 4x the parameters than 1.1, and will be trained on 150-250x the training data! > Expanding on my last note, 1.1 really just taught me how to efficiently compress weights and etc without loss, and laid numerous blueprints I'll re-use for 1.2 PRO. > It's definitely fair to say 1.1 is dumb for a 1.7 million model, which is likely due to it only being trained on 40 million tokens (which isn't really enough for a model of this size), a more efficient training size would've been closer to 1.7-2.55 billion tokens of data. > You're probably asking why the code in the backdrop looks weird, and *why* the code for the AI is in the backdrop? Well, block coding at this scale was too tedious for me, especially when trying to outsource help to debug, so I decided to use Goboscript! It's a very helpful compiler that translates text language to .sb3 files. https://aspiz.uk/goboscript/ide/ ================================ Credits: > Cud v3.5 [@CoderGuy09] | Inspiration | https://scratch.mit.edu/projects/1386471146/ > claude.ai | Helped debug python and goboscripts. > Me | Basically everything
