I'm building Rusty-R2, exploring efficient, post-transformer architectures you can train from scratch on ordinary hardware. Not cloud-dependent, not locked behind paywalls.
The goal: small, customizable, agentic AI that's fully open. Built with open data, trained transparently, AGPL licensed so it stays open forever. Every contributor keeps their copyright.
Right now it's just me working on this, but I'm looking for people who want to build something real together. We're aiming to explore AI safety through transparency, responsible pretraining, and community-driven development, rather than post-training methods that censor or lobotomize the model. These are goals, not finished achievements. We're learning by doing, fig...
I'm building Rusty-R2, exploring efficient, post-transformer architectures you can train from scratch on ordinary hardware. Not cloud-dependent, not locked behind paywalls.
The goal: small, customizable, agentic AI that's fully open. Built with open data, trained transparently, AGPL licensed so it stays open forever. Every contributor keeps their copyright.
Right now it's just me working on this, but I'm looking for people who want to build something real together. We're aiming to explore AI safety through transparency, responsible pretraining, and community-driven development, rather than post-training methods that censor or lobotomize the model. These are goals, not finished achievements. We're learning by doing, figuring this out together.
Current status: Currently using a RWKV-like architecture, but I'm completely open to experimenting with other architectures. Base model trains successfully on consumer hardware the last time I tested, but I've been focused on choosing datasets and haven't tested the training pipeline in a few days (14M parameters, 1000 training steps in ~98 minutes on a single GTX1650TI GPU with 4GB of vram, training actually uses less than 2gb ram/vram combined in its current state). Supervised learning pipeline is working. The model outputs something, but it's not coherent or usable yet. It needs way more data and training time. Agentic fine-tuning layer has module import issues that need fixing. Interactive terminal has protocol errors to debug. Most of the code is AI-generated. I'm a systems administrator, not a developer, so I use AI as a coding tool while I handle the architecture and system design.
This is early development, but the goal is real, usable, agentic models. Not a toy project. The supervised training works, but the agentic components aren't wired up correctly yet, and the base model needs significantly more training. I'm putting this out there for transparency, showing what works and what doesn't, inviting people who want to help solve real problems or just watch the process unfold.
Once we figure out how to produce high quality models, I'd like to make the entire training process as user-friendly and accessible to laypeople as possible.
You don't need to submit code to participate (though contributions are welcome). All contributions are welcome under the project's AGPL license.
If you want to participate but don't like the direction I'm taking it, fork it and do your own thing. That's what open source is for. I maintain the final say in what pull requests do and do not get merged into MY repo of course.
Right now everything is on GitHub. I might set up a Discord or Matrix channel for community discussion later if there's interest. We might also build Jupyter notebooks to make training environments more reproducible, and/or so people could use Kaggle or Colab. We'll see where this goes.