universal_server() ->
receive
{become, F} ->
F()
end.

factorial_server() ->
receive
{From, N} ->
From ! factorial(N),
factorial_server()
end.

factorial(0) -> 1;
factorial(N) -> N * factorial(N-1).

Joe Armstrong wrote a blog post titled My favorite Erlang Program, which showed a very simple universal server written in Erlang:

universal_server() ->
receive
{become, F} ->
F()
end.

You could then write a small program that could fit this function F:

factorial_server() ->
receive
{From, N} ->
From ! factorial(N),
factorial_server()
end.

factorial(0) -> 1;
factorial(N) -> N * factorial(N-1).

If you had an already running universal server, such as you would by having called Pid = spawn(fun universal_server/0), you could then turn that universal server into the factorial server by calling Pid ! {become, fun factorial_server/0}.

Weeds growing in driveway cracks

Joe Armstrong had a way to get to the essence of a lot of concepts and to think about programming as a fun thing. Unfortunately for me, my experience with the software industry has left me more or less frustrated with the way things are, even if the way things are is for very good reasons. I really enjoyed programming Erlang professionally, but I eventually got sidetracked by other factors that would lead to solid, safe software—mostly higher level aspects of socio-technical systems, and I became SRE.

But a part of me still really likes dreaming about the days where I could do hot code loading over entire clusters—see A Pipeline Made of Airbags—and I kept thinking about how I could bring this back, but within the context of complex software teams running CI/CD, building containers, and running them in Kubernetes. This is no short order, because we now have decades of lessons telling everyone that you want your infrastructure to be immutable and declared in code.

I also have a decade of experience telling me a lot of what we’ve built is a frustrating tower of abstractions over shaky ground. I know I’ve experienced better, my day job is no longer about slinging my own code, and I have no pretense of respecting the tower of abstraction itself.

A weed is a plant considered undesirable in a particular situation, "a plant in the wrong place". Examples commonly are plants unwanted in human-controlled settings, such as farm fields, gardens, lawns, and parks. Taxonomically, the term "weed" has no botanical significance, because a plant that is a weed in one context is not a weed when growing in a situation where it is wanted.

Like the weeds that decide that the tiniest crack in a driveway is actually real cool soil to grow in, I’ve decided to do the best thing given the situation and bring back Erlang live code upgrades to modern CI/CD, containerized and kubernetized infrastructure.

If you want the TL:DR; I wrote the dandelion project, which shows how to go from an Erlang/OTP app, automate the generation of live code upgrade instructions with the help of pre-existing tools and CI/CD, generate a manifest file and store build artifacts, and write the necessary configuration to have Kubernetes run said containers and do automated live code upgrades despite its best attempts at providing immutable images. Then I pepper in some basic CI scaffolding to make live code upgrading a tiny bit less risky. This post describes how it all works.

A Sample App

A lot of "no downtime" deployments you’ll find for Kubernetes are actually just rolling updates with graceful connection termination. Those are always worth supporting (even if it can be annoying to get right), but it has a narrow definition of downtime that’s different from what we’re aiming for here: no need to restart the application, dump and re-hydrate the state, nor to drop a single connection.

A small server with persistent connections

I wrote a really trivial application, nothing worth calling home about. It’s a tiny TCP server with a bunch of acceptors where you can connect with netcat (nc localhost 8080) and it just displays stuff. This is the bare minimum to show actual "no downtime": a single instance, changing its code definitions in a way that is directly observable to a client, without ever dropping a connection.

The application follows a standard release structure for Erlang, using Rebar3 as a build tool. Its supervision structure looks like this:

top level
supervisor
/      \
connection   acceptors
supervisor   supervisor
|            |
connection    acceptor
workers        pool
(1 per conn)

The acceptors supervisor starts a TCP listen socket, which is passed to each worker in the acceptor pool. Upon each accepted connection, a connection worker is started and handed the final socket. The connection worker then sits in an event loop. Every second it sends in a small ASCII drawing of a dandelion, and for every packet of data it receives (coalesced or otherwise), it sends in a line containing its version.

A netcat session looks like this:

$ nc localhost 8080

@
\ |
__\!/__


@
\ |
__\!/__

> ping!
vsn: 0.1.5

@
\ |
__\!/__

^C

Universal container: a plan

Modern deployments are often done with containers and Kubernetes. I’m assuming you’re familiar with both of these concepts, but if you want more information—in the context of Erlang—then Tristan Sloughter wrote a great article on Docker and Erlang and another one on using Kubernetes for production apps.

In this post, I’m interested in doing two things:

1. Have an equivalent to Joe Armstrong’s Universal server
2. Force the immutable world to nevertheless let me do live code upgrades

The trick here is deceptively simple, enough to think "that can’t be a good idea." It probably isn’t.

The plan? Use a regular container someone else maintains and just wedge my program’s tarball in there. I can then use a sidecar to automate fetching updates and applying live code upgrades without Kubernetes knowing anything about it.

Erlang Releases: a detour

To understand how this can work, we first need to cover the basics of Erlang releases. A good overview of the Erlang Virtual Machine’s structure is an article I’ve written in OTP at a high level, but that I can summarize here by describing the following layers, from lowest to highest:

• There is an Erlang Runtime System (ERTS), which is essentially the VM itself, written in C. This can’t be live upgraded, and offers features around Erlang’s immutability, preemptive scheduling, memory allocation, garbage collection, and so on.
• A few pre-loaded modules that offer core functionality around files and sockets. I don’t believe that these get to be live upgraded either.
• There is a pair of libraries ("applications" in Erlang-speak) called kernel and stdlib that define the most basic libraries around list processing, distributed programs, and the core of "OTP", the general development framework in the language. These can be live upgraded, but pretty much nobody does that
• Then we have the Erlang standard library. This includes things such as TLS support, HTTP clients and servers, Wx bindings, test frameworks, the compiler, and extra scaffolding around OTP niceties, to name a few.

Your own project is pretty much just your own applications ("libraries"), bundled with select standard library libraries and a copy of the Erlang system:

The end result of this sort of ordeal is that every Erlang project is pretty much people writing their own libraries (in blue in the drawing above), fetching bits from the Erlang base install (in red in the drawing above), and then using tools (such as Rebar3) to repackage everything into a brand new Erlang distribution. A detailed explanation of how this happens is also in Adopting Erlang.

Whatever you built on one system can be deployed on an equivalent system. If you built your app on a 64 bit linux—and assuming you used static libraries for OpenSSL or LibreSSL, or have equivalent ones on a target system—then you can pack a tarball, unpack it on the other host system, and get going. Those requirements don’t apply to your code, only the standard library. If you don’t use NIFs or other C extensions, your own Erlang code, once built, is fully portable.

The cool thing is that Erlang supports making a sort of "partial" release, where you take the Erlang/OTP part (red) and your own apps (blue) in the above image, only package your own app’s and a sort of "figure out the Erlang/OTP part at run-time" instruction, and your application is going to be entirely portable across all platforms (Windows, Linux, BSD, MacOS) and supported architectures (x86, ARM32, ARM64, etc.)

I’m mentioning this because for the sake of this experiment, I’m running things locally on a M1 macbook air (ARM64), with MicroK8s (which runs a Linux/aarch64), but am using Github Actions, which are on an x86 linux. So rather than using a base ubuntu image and then needing to run the same sort of hardware family everywhere down the build chain, I’ll be using an Erlang image from dockerhub to provide the ERTS and stdlib, and will then have the ability to make portable builds from either my laptop or github actions and deploy them onto any Kubernetes cluster—something noticeably nicer than having to deal with cross-compilation in any language.

Controlling Releases

The release definition for Dandelion accounts for the above factors, and looks like this:

{relx, [{release, {dandelion, "0.1.5"}, % my release and its version (dandelion-0.1.5)
[dandelion,                    % includes the 'dandelion' app and its deps
sasl]},                       % and the 'sasl' library, which is needed for
% live code upgrades to work

%% set runtime configuration values based on the environment in this file
{sys_config_src, "./config/sys.config.src"},
%% set VM options in this file
{vm_args, "./config/vm.args"},

%% drop source files and the ERTS, but keep debug annotations
%% which are useful for various tools, including automated
%% live code upgrade plugins
{include_src, false},
{include_erts, false},
{debug_info, keep},
{dev_mode, false}
]}.

The release can be built by calling rebar3 release and packaged by calling rebar3 tar.

Take the resulting tarball, unpack it, and you’ll get a bunch of directories: lib/ contains the build artifact for all libraries, releases/ contains metadata about the current release version (and the structure to store future and past versions when doing live code upgrades), and finally the bin/ directory contains a bunch of accessory scripts to load and run the final code.

Call bin/dandelion and a bunch of options show up:

$ bin/dandelion
Usage: dandelion [COMMAND] [ARGS]

Commands:

foreground              Start release with output to stdout
remote_console          Connect remote shell to running node
rpc [Mod [Fun [Args]]]] Run apply(Mod, Fun, Args) on running node
eval [Exprs]            Run expressions on running node
stop                    Stop the running node
restart                 Restart the applications but not the VM
reboot                  Reboot the entire VM
...
upgrade [Version]       Upgrade the running release to a new version
downgrade [Version]     Downgrade the running release to a new version
install [Version]       Install a release
uninstall [Version]     Uninstall a release
unpack [Version]        Unpack a release tarball
versions                Print versions of the release available
...

So in short, your program’s lifecycle can become:

• bin/dandelion foreground boots the app (use bin/dandelion console to boot a version in a REPL)
• bin/dandelion remote_console pops up a REPL onto the running app by using distributed Erlang

If you’re doing the usual immutable infrastructure, that’s it, you don’t need much more. If you’re doing live code upgrades, you then have a few extra steps:

1. Write a new version of the app
2. Give it some instructions about how to do its live code upgrade
3. Pack that in a new version of the release
4. Put the tarball in releases/
5. Call bin/dandelion unpack $version and the Erlang VM will unpack the new tarball into its regular structure
6. Call bin/dandelion install $version to get the Erlang VM in your release to start tracking the new version (without switching to it)
7. Call bin/dandelion upgrade $version to apply the live code upgrade

And from that point on, the new release version is live.

Hot Code Upgrade Instructions

I’ve sort of papered over the complexity required to "give it some instructions about how to do its live code upgrade." This area is generally really annoying and complex. You first start with appup files, which contain instructions on upgrading individual libraries, which are then packaged into a relup which provides instructions for coordinating the overall upgrade.

If you’re running live code upgrades on a frequent basis you may want to get familiar with these, but most people never bothered, and the vast majority of live code upgrades are done by people writing manual scripts to load specific modules.

A very nice solution that also exists is to use Luis Rascão’s rebar3_appup_plugin which will take two releases, compare their code, and auto-generate instructions on your behalf. By using it, most of the annoyances and challenges are automatically covered for you.

All you need to do is to make sure all versions are adequately bumped, do a few command line invocations, and package it up. This will be a prime candidate for automation soon in this post.

For now though, let’s assume we’ll just put the release in an S3 bucket that the kubernetes cluster has access to, and build our infrastructure on the Kubernetes side.

Universal container: a kubernetes story

Let’s escape the Erlang complexity and don our DevOps hat. We now want to run the code we assume has made it safely to S3. All of it beautifully holds into a single YAML file—which, granted, can’t really be beautiful on its own. I use three containers in a single kubernetes pod:

All of these containers will share a ‘release’ directory, by using an EmptyDir volume. The bootstrap container will fetch the latest release and unpack it there, the dandelion-release container will run it, and the sidecar will be able to interact over the network to manage live code upgrades.

The bootstrap pod runs first, and fetches the first (and current) release from S3. I’m doing so by assuming we’ll have a manifest file (<my-s3-bucket>/dandelion-latest) that contains a single version number that points to the tarball I want ($my*s3*bucket/dandelion-$version.tar.gz). This can be done with a shell script:

#!/usr/bin/env bash
set -euxo pipefail
RELDIR=${1:-/release}
S3_URL="https://${BUCKET_NAME}.s3.${AWS_REGION}.amazonaws.com"
TAG=$(curl "${S3_URL}/${RELEASE}-latest" -s)
wget -nv "${S3_URL}/${RELEASE}-${TAG}.tar.gz" -O "/tmp/${RELEASE}-${TAG}.tar.gz"
tar -xvf "/tmp/${RELEASE}-${TAG}.tar.gz" -C ${RELDIR}
rm "/tmp/${RELEASE}-${TAG}.tar.gz"

This fetches the manifest, grabs the tag, fetches the release, unpacks it, and deletes the old tarball. The dandelion-release container, which will run our main app, can then just call the bin/dandelion script directly:

#!/usr/bin/env bash
set -euxo pipefail
RELDIR=${1:-/release}
exec ${RELDIR}/bin/${RELEASE} foreground

The sidecar is a bit more tricky, but can reuse the same mechanisms. Every time interval (or based on a feature flag or some server-sent signal), check the manifest, and apply the unpacking steps. Something a bit like:

#!/usr/bin/env bash
set -euxo pipefail
RELDIR=${2:-/release}
S3_URL="https://${BUCKET_NAME}.s3.${AWS_REGION}.amazonaws.com"

CURRENT=$(${RELDIR}/bin/${RELEASE} versions | awk '$3=="permanent" && !vsn { vsn=$2 } $3=="current" { vsn=$2 } END { print vsn }')
TAG=$(curl "${S3_URL}/${RELEASE}-latest" -s)
if [[ "${CURRENT}" != "${TAG}" ]]; then
wget -nv "${S3_URL}/${RELEASE}-${TAG}.tar.gz" -O "${RELDIR}/releases/${RELEASE}-${TAG}.tar.gz"
${RELDIR}/bin/${RELEASE} unpack ${TAG}
${RELDIR}/bin/${RELEASE} install ${TAG}
${RELDIR}/bin/${RELEASE} upgrade ${TAG}
fi

Call this in a loop and you’re good to go.

Now here’s the fun bit: ConfigMaps are a Kubernetes thing that lets you take arbitrary metadata, and optionally use them as files into pods. This is how we get close to our universal container.

By declaring the three scripts above as a ConfigMap and mounting them in a /scripts directory, we can then declare the 3 containers in a generic fashion:

initContainers:
- name: dandelion-bootstrap
image: erlang:25.0.2
env:
- ...
volumeMounts:
- name: release
mountPath: /release
- name: scripts
mountPath: /scripts
command:
- /scripts/init-latest.sh
# Regular containers run next
containers:
- name: dandelion-release
image: erlang:25.0.2
env:
- ...
volumeMounts:
- name: release
mountPath: /release
- name: scripts
mountPath: /scripts
command:
- /scripts/boot-release.sh
ports:
- containerPort: 8080
hostPort: 8080
- name: dandelion-sidecar
image: erlang:25.0.2
env:
- ...
volumeMounts:
- name: release
mountPath: /release
- name: scripts
mountPath: /scripts
command:
- /scripts/update-loop.sh

The full file has more details, but this is essentially all we need. You could kubectl apply -f dandelion.yaml and it would get going on its own. The rest is about providing a better developer experience.

Making it Usable

What we have defined now is an expected format and procedure from Erlang’s side to generate code and live upgrade instructions, and a wedge to make this usable within Kubernetes’ own structure. This procedure is somewhat messy, and there are a lot of technical aspects that need to be coordinated to make this usable.

Now comes the time to work around providing a useful workflow for this.

Introducing Smoothver

Semver’s alright. Most of the time I won’t really care about it, though. I’ll go read the changelog and see if whatever I depend on has changed or not. People will pick versions for whichever factor they want, and they’ll very often put a small breaking change (maybe a bug fix!) as non-breaking because there’s an intent being communicated by the version.

Here the semver semantics are not useful. I’ve just defined a workflow that mostly depends on whether the server can be upgraded live or not, with some minor variations. This operational concern is likely to be the main concern of engineers who would work on such an application daily, particularly since as someone deploying and maintaining server-side software, I mostly own the whole pipeline and always consider the main branch to be canonical.

As such, I should feel free to develop my own versioning scheme. Since I’m trying to reorient Dandelion’s whole flow towards continuous live delivery, my versioning scheme should actually reflect and support that effort. I therefore introduce Smoothver (Smooth Versioning):

Given a version number RESTART.RELUP.RELOAD, increment the:

• RESTART version when you make a change that requires the server to be rebooted.
• RELUP version when you make a change that requires pausing workers and migrating state.
• RELOAD version when you make a change that requires reloading modules with no other transformation.

The version number now communicates the relative expected risk of a deployment in terms of disruptiveness, carries some meaning around the magnitude of change taking place, and can be leveraged by tooling.

For example:

• Fixing a bug in a data structure is a RELOAD deploy so long as the internal representation does not change (e.g. swapping a > for a >= in a comparison)
• Adding a new endpoint or route to an existing HTTP API is likely a RELOAD deploy since no existing state relies on it. Rolling it back is a business concern, not a technical one.
• Adding or changing a field in a data structure representing a user is a RELUP operation, since rolling forward or backward implies a data transformation to remain compatible
• Upgrading the VM version is a RESTART because the C code of the VM itself can’t change
• Bumping a stateful dependency that does not provide live code upgrades forces a RESTART version bump

As with anything we do, the version bump may be wrong. But it at least carries a certain safety level in letting you know that a RESTART live code upgrade should absolutely not be attempted.

Engineers who get more familiar with live code upgrades will also learn some interesting lessons. For example, a RELUP change over a process that has tens of thousands of copies of itself may take a long long time to run and be worse than a rolling upgrade. An interesting thing you can do then is turn RELUP changes (which would require calling code change instructions) into basic code reloads by pattern matching an old structure and converting it on each call, turning it into a somewhat stateless roll-forward affair.

That’s essentially converting operational burdens into dirtier code, but this sort of thing is something you do all the time with database migrations (create a new table, double-write, write only to the new one, delete the old table) and that can now be done with running code.

For a new development workflow that tries to orient itself towards live code upgrades, Smoothver is likely to carry a lot more useful information than Semver would (and maybe could be nice for database migrations as well, since they share concerns).

Publishing the Artifacts

I needed to introduce the versioning mechanism because the overall publication workflow will obey it. If you’re generating a new release version that requires a RESTART bump, then don’t bother generating live code upgrade instructions. If you’re generating anything else, do include them.

I’ve decided to center my workflow around git tags. If you tag your release v1.2.3, then v1.2.4 or v1.4.1 all do a live code upgrade, but v2.0.0 won’t, regardless of which branch they go to. The CI script is not too complicated, and is in three parts:

1. Fetch the currently deployed manifest, and see if the newly tagged version requires a live code upgrade ("relup") or not
2. Build the release tarball with the relup instructions if needed. Here I rely purely on Luis’s plugin to handle all the instructions.
3. Put the files on S3

That’s really all there is to it. I’m assuming that if you wanted to have more environments, you could setup gitops by having more tags (staging-v1.2.3, prod-v1.2.5) and more S3 buckets or paths. But everything is assumed to be driven by these builds artifacts.

A small caveat here is that it’s technically possible to generate upgrade instructions (appup files) that map from many to many versions: how to update to 1.2.0 from 1.0.0, 1.0.1, 1.0.2, and so on. Since I’m assuming a linear deployment flow here, I’m just ignoring that and always generating pairs from "whatever is in prod" to "whatever has been tagged". There are obvious race conditions in doing this, where two releases generated in parallel can specify upgrade rules from a shared release, but could be applied and rolled out in a distinct order.

Using the Manifest and Smoothver

Relying on the manifest and versions is requiring a few extra lines in the sidecar’s update loop. They look at the version, and if it’s a RESTART bump or an older release, they ignore it:

# Get the running version
CURRENT=$(${RELDIR}/bin/${RELEASE} versions | awk '$3=="permanent" && !vsn { vsn=$2 } $3=="current" { vsn=$2 } END { print vsn }')
TAG=$(curl "${S3_URL}/${RELEASE}-latest" -s)
if [[ "${CURRENT}" != "${TAG}" ]]; then
IS_UPGRADE=$(echo "$TAG $CURRENT" | awk -vFS='[. ]' '($1==$4 && $2>$5) || ($1==$4 && $2>=$5 && $3>$6) {print 1; exit} {print 0}')
if [[ $IS_UPGRADE -eq 1 ]]; then
wget -nv "${S3_URL}/${RELEASE}-${TAG}.tar.gz" -O "${RELDIR}/releases/${RELEASE}-${TAG}.tar.gz"
${RELDIR}/bin/${RELEASE} unpack ${TAG}
${RELDIR}/bin/${RELEASE} install ${TAG}
${RELDIR}/bin/${RELEASE} upgrade ${TAG}
fi
fi

There’s some ugly awk logic, but I wanted to not host images. The script could be made a lot more solid by looking at whether we’re bumping from the proper version to the next one, and in this it shares a sort of similar race condition to the generation step.

On the other hand, the install step looks at the specified upgrade instructions and will refuse to apply itself (resulting in a sidecar crash) if a bad release is applied.

I figure that alerting on crashed sidecars could be used to drive further automation to ask to delete and replace the pods, resulting in a rolling upgrade. Alternatively, the error itself could be used to trigger a failure in liveness and/or readiness probes, and force-automate that replacement. This is left as an exercise to the reader, I guess. The beauty of writing prototypes is that you can just decide this to be out of scope and move on, and let someone who’s paid to operationalize that stuff to figure out the rest.

Oh and if you just change the Erlang VM’s version? That changes the kubernetes YAML file, and if you’re using anything like helm or some CD system (like ArgoCD), these will take care of running the rolling upgrade for you. Similarly, annotating the chart with a label of some sort indicating the RESTART version will accomplish the same purpose.

You may rightfully ask whether it is a good idea to bring mutability of this sort to a containerized world. I think that using S3 artifacts isn’t inherently less safe than a container registry, dynamic feature flags, or relying on encryption services or DNS records for functional application state. I’ll leave it at that.

Adding CI validation

Versioning things is really annoying. Each OTP app and library, and each release needs to be versioned properly. And sometimes you change dependencies and these dependencies won’t have relup instructions available but you didn’t know and that would break your live code upgrade.

What we can do is add a touch of automation to catch the most obvious failure situations and warn developers early about these issues. I’ve done so by adding a quick relup CI step to all pull requests, by using a version check script that encodes most of that logic.

The other thing I started experimenting with was setting up some sort of test suite for live code upgrades:

# This here step is a working sample, but if you were to run a more
# complex app with external dependencies, you'd also have to do a
# more intricate multi-service setup here, e.g.:
# https://github.com/actions/example-services
- name: Run relup application
working-directory: erlang
run: |
mkdir relupci
tar -xvf "${{ env.OLD_TAR }}" -C relupci
# use a simple "run the task in the background" setup
relupci/bin/dandelion daemon
TAG=$(echo "${{ env.NEW_TAR }}"  | sed -nr 's/^.*([0-9]+\.[0-9]+\.[0-9]+)\.tar\.gz$/\1/p')
cp "${{ env.NEW_TAR }}" relupci/releases/
relupci/bin/dandelion unpack ${TAG}
relupci/bin/dandelion install ${TAG}
relupci/bin/dandelion upgrade ${TAG}
relupci/bin/dandelion versions

The one thing that would make this one a lot cooler is to write a small extra app or release that runs in the background while the upgrade procedure goes on. It could do things like:

• generate constant load
• run smoke tests on critical workflows
• maintain live connections to show lack of failures
• report on its state on demand

By starting that process before the live upgrade and questioning it after, we could ensure that the whole process went smoothly. Additional steps could also look at logs to know if things were fine.

The advantage of adding CI here is that each pull request can take measures to ensure it is safely upgradable live before being merged to main, even if none of them are deployed right away. By setting that gate in place, engineers are getting a much shorter feedback loop asking them to think about live deployments.

Running through a live code upgrade

I’ve run through a few iterations to test and check everything. I’ve set up microk8s on my laptop, ran kubectl -f apply dandelion.yaml and showed that the pod was up and running fine:

$ kubectl -n dandelion get pods
NAME                                    READY   STATUS    RESTARTS   AGE
dandelion-deployment-648db88f44-49jl8   2/2     Running   0          25H

It is possible to run into one of the containers, log on onto a REPL, and see what is going on:

$ kubectl -n dandelion exec -i -t dandelion-deployment-648db88f44-49jl8 -c dandelion-sidecar -- /bin/bash
root@dandelion-deployment-648db88f44-49jl8:/# /release/bin/dandelion remote_console
Erlang/OTP 25 [erts-13.0.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit]

Eshell V13.0.2  (abort with ^G)
(dandelion@localhost)1> release_handler:which_releases().
[{"dandelion","0.1.5",
["kernel-8.4.1","stdlib-4.0.1","dandelion-0.1.5","sasl-4.2"],
permanent},
{"dandelion","0.1.4",
["kernel-8.4.1","stdlib-4.0.1","dandelion-0.1.4","sasl-4.2"],
old}]

This shows that the container had been running for a day, and already had two releases—it first booted on version 0.1.4 and had already gone through a bump to 0.1.5. I ran a small pull request changing the display (and messed up versioning, which CI caught!), merged it, tagged it v0.1.6, and started listening to my Kubernetes cluster:

$ nc 192.168.64.2 8080

@
\ |
__\!/__

...

@
\ |
__\!/__

vsn?
vsn: 0.1.5

@
\ |
__\!/__

*
@
\ |
__\!/__

vsn?
vsn: 0.1.6
*
@
\ |
__\!/__

...

This shows me interrogating the app (vsn?) and getting the version back, and without dropping the connection, having a little pappus floating in the air.

My REPL session was still live in another terminal:

(dandelion@localhost)2> release_handler:which_releases().
[{"dandelion","0.1.6",
["kernel-8.4.1","stdlib-4.0.1","dandelion-0.1.6","sasl-4.2"],
permanent},
{"dandelion","0.1.5",
["kernel-8.4.1","stdlib-4.0.1","dandelion-0.1.5","sasl-4.2"],
old},
{"dandelion","0.1.4",
["kernel-8.4.1","stdlib-4.0.1","dandelion-0.1.4","sasl-4.2"],
old}]

showing that the old releases are still around as well. And here we have it, an actual zero-downtime deploy in a kubernetes container.

Conclusion

Joe’s favorite program could hold on a business card. Mine is maddening. But I think this is because Joe didn’t care for the toolchains people were building and just wanted to do his thing. My version reflects the infrastructure we have put in place, and the processes we want and need for a team.

Rather than judging the scaffolding, I’d invite you to think about what would change when you start centering your workflow around a living system.

Those of you who have worked with bigger applications that have a central database or shared schemas around network protocols (or protobuf files or whatever) know that you approach your work differently when you have to consider how it’s going to be rolled out. It impacts your tooling, how you review changes, how you write them, and ultimately just changes how you reason about your code and changes.

In many ways it’s a more cumbersome way to deploy and develop code, but you can also think of the other things that change: what if instead of having configuration management systems, you could hard-code your config in constants that just get rolled out live in less than a minute—and all your configs were tested as well as your code? Since all the release upgrades implicitly contain a release downgrade instruction set, just how much faster could you rollback (or automate rolling back) a bad deployment? Would you be less afraid of changing network-level schema definitions if you made a habit of changing them within your app? How would your workflow change if deploying took half-a-second and caused absolutely no churn nor disruption to your cluster resources most of the time?

Whatever structure we have in place guides a lot of invisible emergent behaviour, both in code and in how we adjust ourselves to the structure. Much of what we do is a tacit response to our environment. There’s a lot of power in experimenting alternative structures, and seeing what pops up at the other end. A weed is only considered as such in some contexts. This is a freak show of a deployment mechanism, but it sort of works, and maybe it’s time to appreciate the dandelions for what they can offer.