Standards are great, until they aren’t

Implementing a WebDAV/CalDAV client and server should be easy! It’s a well documented spec, standardized in the early 00s, and somewhat widely supported. At least, that’s the naive assumption we started from when creating one for Homechart.

Existing Go Implementations

Now before you mention NIH syndrome, yes, we looked at the existing Go implementation, go-webdav. This library was lacking some key features we needed, like server-side collection synchronization, and the interfaces didn’t really align with our data model. This is also going to be a key feature of our product, so we should have some level of ownership for what gets implemented.

To start creating our client and server, we should read the RFCs, right? Well, where do you start?

How about the original, RFC 2518? Ah, looks like it was somewhat superseded by RFC 4918, but we’re not going to tell you which parts! How about those extension RFCs? There’s only 7 of them…

Reading through the RFCs, all that our implementation cares about is CRUD for Calendar events. After spending almost a month trying to implement the full RFC spec, we threw in the towel, there’s just to much legacy cruft that we didn’t need.

Reverse Engineering

With a decent understanding of the RFC in hand, we instead looked into reverse engineering existing clients and servers by inspecting their requests and responses. This process was MUCH faster, and we quickly had the API mapped out and what kind of requests/responses we needed to support.

We started by identifying the clients/servers we wanted to support:

Clients:

• Apple Calendar
• DavX
• Thunderbird

Servers:

• Apple iCloud
• Google Calendar
• Radicale

And then ran HTTP proxies or WireGuard to capture the HTTP requests. Because WebDAV is so obtuse, you not only need to inspect the HTTP body, but also the headers!

XML in Go

As an aside, we spent quite a bit of time trying to make XML work well in Go. The default Go XML library is truly terrible, and we decided to create a wrapper around it for managing XML nodes similar to how JavaScript manages HTML nodes:

var davDisplayName = xmel.Element{
Name:  "displayname",
Space: davNS,
}

davDisplayName.SetValue("name")
n, err := davResponse.Find(davCollectionType)
davOwner = davOwner.AddChild(davHref.SetValue("http://example.com"))

With WebDAV having such an…“unstructured” schema to a lot of the requests/responses, this library was key in helping us marshal/unmarshal things without writing a bunch of “best case” structs.

Standards are Just Suggestions

When we finally had our MVP built out, we put it to the test: validating our client and server against the existing implementations! For the most part, it worked as expected, but as always, things drift from the RFC.

Apple and Google, for instance, don’t implement half of the RFCs, and basically provide a MVP for other clients to use. They don’t really document what they support/don’t support, as WebDAV is supposed to do it via HTTP responses advertising capabilities, but both seem to provide generic responses advertising capabilities they don’t have a lot of the time.

The clients were another story. CalDAV clients are all over the place with what they support and how they will request it. Most clients should prefer to support sync-collection as it’s very efficient, but Apple Calendar doesn’t, and uses ctags and etags instead.

As a little fish in a big pond, it’s frustrating dealing with situations where big providers can skirt around some standards or add quirks for their implementations, but I’m required to follow them to the T because I don’t have their inertia. I can’t file a bug, or a lawsuit, against them claiming nonconformance, they’ll tell me to get bent. And you see this in other open source libraries too, where they’re littered with comments about workarounds for Google’s specific implementation or whatever.

I wouldn’t recommend anyone who values their sanity to pursue creating a WebDAV/CalDAV library.