I don’t like RSS readers. I know, this is blasphemous especially on a website where I’m actively encouraging you to subscribe through RSS. As someone writing stuff, RSS is great for me. I don’t have to think about it, the requests are pretty light weight, I don’t need to think about your personal data or what client you are using. So as a protocol RSS is great, no notes.
However as something I’m going to consume, it’s frankly a giant chore. I feel pressured by RSS readers, where there is this endlessly growing backlog of things I haven’t read. I rarely want to read all of a websites content from beginning to end, instead I like to jump between them. I also don’t really care if the content is chronological, like an old post about something interesting isn’t less compelling to m…
I don’t like RSS readers. I know, this is blasphemous especially on a website where I’m actively encouraging you to subscribe through RSS. As someone writing stuff, RSS is great for me. I don’t have to think about it, the requests are pretty light weight, I don’t need to think about your personal data or what client you are using. So as a protocol RSS is great, no notes.
However as something I’m going to consume, it’s frankly a giant chore. I feel pressured by RSS readers, where there is this endlessly growing backlog of things I haven’t read. I rarely want to read all of a websites content from beginning to end, instead I like to jump between them. I also don’t really care if the content is chronological, like an old post about something interesting isn’t less compelling to me than a newer post.
What I want, as a user experience, is something akin to TikTok. The whole appeal of TikTok, for those who haven’t wasted hours of their lives on it, is that I get served content based on an algorithm that determines what I might think is useful or fun. However what I would like is to go through content from random small websites. I want to sit somewhere and passively consume random small creators content, then upvote some of that content and the service should show that more often to other users. That’s it. No advertising, no collecting tons of user data about me, just a very simple “I have 15 minutes to kill before the next meeting, show me some random stuff.”
In this case the “algorithm” is pretty simple: if more people like a thing, more people see it. But with Google on its way to replacing search results with LLM generated content, I just wanted to have something that let me play around with the small web the way that I used to.
There actually used to be a service like this called StumbleUpon which was more focused on pushing users towards popular sites. It has been taken down, presumably because there was no money in a browser plugin that sent users to other websites whose advertising you didn’t control.
TL;DR
You can go download the Firefox extension now and try this out and skip the rest of this if you want. https://timewasterpro.xyz/ If you hate it or find problems, let me know on Mastodon. https://c.im/@matdevdug
Functionality
So I wanted to do something pretty basic. You hit a button, get served a new website. If you like the website, upvote it, otherwise downvote it. If you think it has objectionable content then hit report. You have to make an account (because I couldn’t think of another way to do it) and then if you submit links and other people like it, you climb a Leaderboard.
On the backend I want to (very slowly so I don’t cost anyone a bunch of money) crawl a bunch of RSS feeds, stick the pages in a database and then serve them up to users. Then I want to track what sites get upvotes and return those more often to other users so that “high quality” content shows up more often. “High quality” would be defined by the community or just me if I’m the only user.
It’s pretty basic stuff, most of it copied from tutorials scattered around the Internet. However I really want to drive home to users that this is not a Serious Thing. I’m not a company, this isn’t a new social media network, there are no plans to “grow” this concept beyond the original idea unless people smarter than me ping with me ideas. So I found this amazing CSS library: https://sakofchit.github.io/system.css/
The Apple’s System OS design from the late-80s to the early 90s was one of my personal favorites and I think would send a strong signal to a user that this is not a professional, modern service.
Great, the basic layout works. Let’s move on!
Backend
So I ended up doing FastAPI because it’s very easy to write. I didn’t want to spend a ton of time writing the API because I doubt I nailed the API design on the first round. I use sqlalchemy for the database. The basic API layout is as follows:
- admin - mostly just generating read-only reports of like “how many websites are there”
- leaderboard - So this is my first attempt at trying to get users involved. Submit a website that other people like? Get points, climb leaderboard.
The source for the RSS feeds came from the (very cool) Kagi small web Github. https://github.com/kagisearch/smallweb. Basically I assume that websites that have submitted their RSS feeds here are cool with me (very rarely) checking for new posts and adding them to my database. If you want the same thing as this does, but as an iFrame, that’s the Kagi small web service.
The scraping work is straightforward. We make a background worker, they grab 5 feeds every 600 seconds, they check for new content on each feed and then wait until the 600 seconds has elapsed to grab 5 more from the smallweb list of RSS feeds. Since we have a lot of feeds, this ends up look like we’re checking for new content less than once a day which is the interval that I want.
Then we write it out to a sqlite database and basically track “has this URL been reported”, if so, put it into a review queue and then how many times this URL has been liked or disliked. I considered a “real” database but honestly sqlite is getting more and more scalable every day and its impossible to beat the immediate start up and functionality. Plus very easy to back up to encrypted object storage which is super nice for a hobby project where you might wipe the prod database at any moment.
In terms of user onboarding I ended up doing the “make an account with an email, I send a link to verify the email”. I actually hate this flow and I don’t really want to know a users email. I never need to contact you and there’s not a lot associated with your account, which makes this especially silly. I have a ton of email addresses and no real “purpose” in having them. I’d switch to Login with Apple, which is great from a security perspective but not everybody has an Apple ID.
I also did a passkey version, which worked fine but the OSS passkey handling was pretty rough still and most people seem to be using a commercial service that handled the “do you have the passkey? Great, if not, fall back to email” flow. I don’t really want to do a big commercial login service for a hobby application.
Auth is a JWT, which actually was a pain and I regret doing it. I don’t know why I keep reaching for JWTs, they’re a bad user experience and I should stop.
Can I just have the source code?
I’m more than happy to release the source code once I feel like the product is in a somewhat stable shape. I’m still ripping down and rewriting relatively large chunks of it as I find weird behavior I don’t like or just decide to do things a different way.
In the end it does seem to do whats on the label. We have over 600,000 individual pages indexed.
So how is it to use?
Honestly I’ve been pretty pleased. But there are some problems.
First I couldn’t find a reliable way of switching the keyboard shortcuts to be Mac/Windows specific. I found some options for querying platform but they didn’t seem to work, so I ended up just hardcoding them as Alt which is not great.
The other issue is that when you are making an extension, you spend a long time working with these manifests.json. The specific part I really wasn’t sure about was:
"browser_specific_settings": {
"gecko": {
"id": "[email protected]",
"strict_min_version": "80.0",
"data_collection_permissions": {
"required": ["authenticationInfo"]
}
}
}
I’m not entirely sure if that’s all I’m doing? I think so from reading the docs.
Anyway I built this mostly for me. I have no idea if anybody else will enjoy it. But if you are bored I encourage you to give it a try. It should be pretty light weight and straight-forward if you crack open the extension and look at it. I’m not loading any analytics into the extension so basically until people complain about it, I don’t really know if its going well or not.
Future stuff
- I need to sort stuff into categories so that you get more stuff in genres you like. I don’t 100% know how to do that, maybe there is a way to scan a website to determine the “types” of content that is on there with machine learning? I’m still looking into it.
- There’s a lot of junk in there. I think if we reach a certain number of downvotes I might put it into a special “queue”.
- I want to ensure new users see the “best stuff” early on but there isn’t enough data to determine “best vs worst”.
- I wish there were more independent photography and science websites. Also more crafts. That’s not really a “future thing”, just me putting a hope out into the universe. Non-technical beta testers get overwhelmed by technical content.