A 50M parameter large language model made in Pytorch! Can be locally trained, inferenced and tinkered with, and comes with a fun demo (more to come) that generates wikipedia pages. Modelfiles can be found on demo pages as they are too large for Github.
Tinyapper's pretrained generator completes a heading, generating a fake Wikipedia article. Try it out at https://yapperpedia.craisin.tech!
Developed with Python 3.13, use older versions at your own risk:
git clone https://github.com/craisined/tinyapper.git
pip install -r requirements.txt # use a virtual environment if desiredThis hosts a demo server on port 8000:
cd demos/yapperpedia
curl -o static/yapperpedia.pt https://yapperpedia.craisin.tech/static/yapperpedia.pt
python3 app.pyThis is still basically disfunctional without gpu, since KV cache hasn't been implemented yet. A container of the demo is published at ghcr.io/craisined/tinyapper for those with GPU access.
For local training / inference, start with local.ipynb. If more flexibility is desired, take a gander through infer.py and train.py. Finally, all model architecture stuff is in model.py. Enjoy!
Tinyapper uses a transformer based architecture with 12 transformer layers. Each layer is comprised of an 8 headed multi attention layer along with a feed forward network. GeLU is used throughout as an activation. GPT-2s tokenizer was used for tokenization, and AdamW was used for optimization.
Wikitext-103-raw-v1 was used as a dataset.
- AI usage was kept to research, debugging, and boilerplate code, as I am trying to work on my Actual Intellegence. An exception was made for RegEx parsing because RegEx stinks
- Pull requests are appreciated if you ever improve on this!
- The demo is currently hosted on a CPU only machine D: If I undergo enough boredom/support, future iterations might be on the cloud :O
- This project was made for Hackclub's Stardance! Feel free to leave a good rating to fund my cloud credit :D