Back in 1997, programmer and open-source activist Eric Raymond wrote an online essay, The Cathedral and the Bazaar, making the case for open-source software development, using Linux as an example:

Linux is subversive. Who would have thought even five years ago (1991) that a world-class operating system could coalesce as if by magic out of part-time hacking by several thousand developers scattered all over the planet, connected only by the tenuous strands of the Internet? . . . Linux overturned much of what I thought I knew. I had been preaching the Unix gospel of small tools, rapid prototyping and evolutionary programming for years. But I also believed there was a certain critical complexity above which a more centralized, a priori approach was required. I believed that the most important software (operating systems and really large tools like the Emacs programming editor) needed to be built like cathedrals, carefully crafted by individual wizards or small bands of mages working in splendid isolation, with no beta to be released before its time.

Linus Torvalds’s style of development–release early and often, delegate everything you can, be open to the point of promiscuity–came as a surprise. No quiet, reverent cathedral-building here–rather, the Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches (aptly symbolized by the Linux archive sites, who’d take submissions from anyone) out of which a coherent and stable system could seemingly emerge only by a succession of miracles.

The fact that this bazaar style seemed to work, and work well, came as a distinct shock. . . .

Lots has happened in software workflow in the past quarter century, and it’s my impression that developers try to get the best of the cathedral and the bazaar. Ultimately, the two forms go together: any “bazaar” is built from individual “stalls” or mini-cathedrals, while any “cathedral” must ultimately compete in a larger “marketplace” or bazaar. So the question isn’t really “the cathedral or the bazaar,” but rather, where should development be more cathedral-like and where should it be bazaar-like. Raymond’s article was helpful in framing development problems in terms of these metaphors, and it connects closely to ideas in economics regarding the roles of firms within the larger economy.

The two software products that I use the most are R and Stan, and I’d say that R has suffered from being too bazaar-like and Stan has suffered from being too cathedral-like.

R code can be a mess because there’s no standard way that things are written, and also because it keeps adding layers and layers of complexity. Back in the day, if you had a matrix, it was a matrix and you could display it access its entries directly. Nowadays, I often need to do tricks such as as.numeric() or as.matrix() to strip away various structures in order to get at the actual numbers. I’m not saying this is always bad–I’m sure that the bazaar-keepers of R have good reasons for all this–it has just become more and more confusing to access attributes, objects, data, whatever.

When I say that Stan is too cathedral-like, I just mean that it seems to take a long time for improvements in the algorithms to show up in the program. I understand that this is important for quality control, I just find it frustrating to not be able to easily access the latest developments.

For both R and Stan, I’m not saying there’s some easy alternative. Things are what they are for a reason. I’m just using these as examples of how I sometimes think in terms of cathedrals and bazaars.

Statistical workflow

The most important difference between statistical workflow and mere statistical inference is that workflow is all about building trust. Steps such as simulation-based model checking, predictive model evaluation, and comparison of the results of multiple models are ways to stress-test our fitted models, to reveal their weaknesses and, sometimes, to lead to more confidence in the results.

It’s possible to fit models with all the workflow, but then you can end up with wrong answers, such as the ridiculous estimate that losing an election for governor costs 5-10 years of life.

One way to think of this is to consider medieval cathedrals. Still standing after all those centuries–that’s so impressive! But there’s selection. These cathedrals were built without good theory, and lots of them collapsed. As much as possible we would like to use science and engineering workflow to avoid building cathedrals that will immediately crumble. Science is a kind of cathedral–a collective human endeavor with large goals–and we would like to design it better.

But science is also a kind of bazaar . . . and, here’s the point: Bazaars, like cathedrals, are designed. They’re not designed in the same way as cathedrals, but they don’t just happen either. There have to be rules, to keep the bazaar from becoming a bloodbath. It’s fair enough to think of the current system of science, with its millions of scientists and hundreds of thousands of research labs, as being a sort of bazaar, and we’re trying to improve the workflow, which would make the bazaar more effective. I think we can do better than the current system of p-values, failed replications, deterministic claims, and waste of reviewer resources.