As most of my readers are aware, I ported FreeBSD to run on the Amazon EC2 cloud, and have been maintaining the platform ever since. Between this and my (far more recent) role as FreeBSD release engineering lead, I sometimes get a question from other cloud providers: “What’s involved in supporting FreeBSD” — or sometimes, “How can we make FreeBSD on our cloud work as well as FreeBSD works on EC2?”. This came up again while I was at the FreeBSD Vendor Summit in San Jose, and it occurred to me that it would be useful to write things down once — even if there’s only a handful of clouds which aspire to compete with Amazon, it will save me time and allow me to give a more…
As most of my readers are aware, I ported FreeBSD to run on the Amazon EC2 cloud, and have been maintaining the platform ever since. Between this and my (far more recent) role as FreeBSD release engineering lead, I sometimes get a question from other cloud providers: “What’s involved in supporting FreeBSD” — or sometimes, “How can we make FreeBSD on our cloud work as well as FreeBSD works on EC2?”. This came up again while I was at the FreeBSD Vendor Summit in San Jose, and it occurred to me that it would be useful to write things down once — even if there’s only a handful of clouds which aspire to compete with Amazon, it will save me time and allow me to give a more complete answer than if I’m trying to answer off the top of my head at a conference.
1. Contact us. It may sound obvious, but we can’t help you if you don’t get in touch with us. Perhaps slightly less obvious: Please email the release engineering team (re@freebsd.org). Don’t just email me directly; in addition to the fact that additional eyes from the release engineering team make it less likely that I’ll miss an email, I’m not going to be the release engineering lead forever; I try to CC the team on as much as possible so that whoever succeeds me will have the maximum possible opportunity to see what’s involved.
2. We’ll need a sponsored account. In AWS I actually have three sponsored accounts — the account I use for development, the release engineering account (for security reasons, this needs to be an account which is only used for publishing images), and an account which hosts some FreeBSD project infrastructure. The first and third are optional, but you want us to publish images and we don’t want to pay to do this. (Why should you want us to publish images, rather than doing it yourself? Among other things, because that dramatically increases the odds that I can write “is available” in a release announcement rather than “will hopefully be available soon”.)
3. Tell us how to publish images. Hopefully you’ve already determined that FreeBSD can run in your cloud; but we need to know how to package it up (do you need a raw disk image, or something more complex?), upload it into your cloud, and often as a third step convert the uploaded disk image into whatever your version of an “AMI” is. We would like to integrate this into the release building process and publish images as part of our weekly snapshot build process; experience shows that this works much better than only uploading images when a release arrives.
4. Find someone who can be a liaison between you and the FreeBSD project. The ideal person here is a FreeBSD committer who works for your company. It’s important that they are able to contact the right people in your company when issues arise, and it’s essential that they’re sufficiently involved in FreeBSD to understand the project culture and be aware of ongoing events (e.g. release cycles).
5. Find someone who will test the weekly images. At a minimum, this means making sure the images exist, boot, and pass some minimal testing every week. Things break, often for wildly unexpected reasons; detecting and reporting problems promptly is important. The person who does this doesn’t need to be a developer, although some basic shell scripting ability is certainly useful; the most important thing is that they are reliable and diligent. This is probably a couple hours a week on average, but that is a bit misleading; it’s more like an hour a week if nothing breaks, but five hours a week if something goes wrong and needs to be investigated. It’s ideal if this is your FreeBSD liaison, since once they find problems they’ll need to talk to people to get them fixed, but it’s not absolutely necessary. You may be tempted to just run automated QA. That’s great, but I don’t think it can replace a human poking at virtual machines; my experience with EC2 is that images break in all sorts of wild ways, and nothing beats a human for saying “huh, I have no idea what’s going on here but something seems weird”.
6. Make sure your FreeBSD liaison is aware of upcoming hardware well in advance. You’re probably doing new “generations” of cloud hardware every two years, and you probably only have it ready for testing a couple months before it launches. But each FreeBSD stable branch only gets a release every 6 months (two branches alternating means a release every quarter); if you wait until you have hardware available, there won’t be a release with any changes you need on launch day. A lot of useful planning can happen before hardware is available, but only if there’s someone who is aware of what’s coming and is in a position to say “hey, do we support foo in our bar driver?”
7. Provide drivers for any special hardware you have. This probably means network interfaces, since we seem to be moving towards a world where everyone uses nvme disks for storage but everyone has their own custom network hardware. Here you’re going to need a kernel developer with some experience with FreeBSD, but assuming you wrote a Linux driver, a lot of that work can be reused as long as you don’t mind putting a BSD license on it. Ideally your driver maintainer will get a FreeBSD commit bit and maintain it in our tree — we have a process for giving commit privileges to driver maintainers who are employed by vendors — if this isn’t an option, you’ll need your “FreeBSD liaison” to push driver updates in for you. There is absolutely no need for the person maintaining your network driver to be the person doing weekly testing.
That’s the bare minimum to have a functional FreeBSD offering. EC2 goes beyond this; they spent a year sponsoring me to work part-time on FreeBSD and got neat features like boot performance plots and multiple AMI “flavours” out, and I’ve also done a lot of things over the years as unfunded work aimed at improving FreeBSD on EC2 — for example, “install security updates when we first boot” and “make FreeBSD boot fast” — which can also yield benefits for FreeBSD on other clouds.
It’s going to take a lot of work to make any other cloud’s FreeBSD support as good as EC2’s FreeBSD support. I intend to keep it that way. But I’d like to have more clouds at least providing functional FreeBSD offerings — and to put in the work needed to keep them that way.