Tuesday, 20 January 2026

Web and Internet

“The Open Web” means several things to different people, depending on context, but recently discussions have focused on the Web’s Openness in terms of access to information – how easy it is to publish and obtain information without barriers there.

David Schinazi and I hosted a pair of ad hoc sessions on this topic at the last IETF meeting in Montreal and the subsequent W3C Technical Plenary in Kobe; you can see the notes and summaries from those sessions. This post contains my thoughts on the topic so far, after some simmering.

The Open Web is Amazing

For most of human history, it’s been difficult to access information. As an average citizen, you had to work pretty hard to access academic texts, historical writings, literature, news, public information, and so on. Libraries were an amazing innovation, but locating and working with the information there was still a formidable challenge.

Likewise, publishing information for broad consumption required resources and relationships that were unavailable to most people. Gutenberg famously broke down some of those barriers, but many still remained: publishing and distributing books (or articles, music, art, films) required navigating extensive industries of gatekeepers, and often insurmountable costs and delays.

Tim Berners-Lee’s invention cut through all of that; it was now possible to communicate with the whole world at very low cost and almost instantaneously. Various media industries were disrupted (but not completely displaced) by this innovation, and reinterpreted roles for intermediaries (e.g., search engines for librarians, online marketplaces for ‘brick and mortar’ shops) were created.

Critically, a norm was also created; an expectation that content was easy to access, didn’t require paying or logging in. This was not enforced, and it was not always honoured: there were still subscription sites, and that’s OK, but they didn’t see the massive network effects that hyperlinks and browsers brought.

It is hard to overstate the benefits of this norm. Farmers in developing countries now have easy access to guidelines and data that help their crops succeed. Students around the world have access to resources that were unimaginable even a few decades ago. They can also contribute to that global commons of content, benefiting others as they build a reputation for themselves.

The Open Web is an amazing public good, both for those who consume information and those who produce it. By reducing costs and friction on both sides, it allows people all over the world to access and create information in a way – and with an ease – that would have been unimaginable to our predecessors. It’s worth fighting for.

People Have Different Motivations for Opening Content

We talk about “The Open Web” in the singular, but in fact there are many motivations for making content available freely online.

Some people consciously make their content freely available on the Web because they want to contribute to the global commons, to help realise all of the benefits described above.

Many don’t, however.

Others do it because they want to be discovered and build a reputation. Or because they want to build human connections. Or because they want revenue from putting ads next to the content. Or because they want people to try their content out and then subscribe to it on the less-than-open Web.

Most commonly, it’s a blend of many (or even all) of these motivations.

Discussions of the Open Web need to consider all of them distinctly – what about their environments are changing, and what might encourage or discourage different kinds of Open Web publishers. Only focusing on some motivations or creating “purity tests” for content isn’t helpful.

There are Many Kinds of “Open”

Likewise, there are many kinds of “open.” While some Open Web content doesn’t come with any strings, much of it does. You might have to allow tracking for ads. While an article might be available to search engines (to drive traffic), you might have to register for an account to view the content as an individual.

There are serious privacy considerations associated with both of these, but those concerns should be considered as distinct from those regarding open access to information. People sometimes need to get a library card to access information at their local library (in person or online), but that doesn’t make the information less open.

One of the most interesting assertions at the meetings we held was about advertising-supported content: that it was more equitable than “micro-transactions” and similar pay-to-view approaches, because it makes content available to those who would otherwise not be able to afford it.

At the same time, these ‘small’ barriers – for example, requirements to log in after reading three articles – add up, reducing the openness of the content. If the new norm is that everyone has to log in everywhere to get Web content (and we may be well on our way to that), the Open Web suffers.

Similarly, some open content is free to all comers and can be reused at will, where other examples have technical barriers (such as bot blockers or other selective access schemes) and/or legal barriers (namely, copyright restrictions).

It Has to be Voluntary

Everyone who publishes on the Open Web does so because they want to – because the benefits they realise (see above) outweigh any downsides.

Conversely, any content not on the Open Web is not there because the owner has made the judgement that it is not worthwhile for them to do so. They cannot be forced to “open up” that content – they can only be encouraged.

Affordances and changes in infrastructure, platforms, and other aspects of the ecosystem – sometimes realised in technical standards, sometimes not – might change that incentive structure and create the conditions for more or less content on the Open Web. They cannot, however, be forced or mandated.

To me, this means that attempts to coerce different parties into desired behaviors are unlikely to succeed – they have to want to provide their content. That includes strategies like withholding capabilities from them; they’ll just go elsewhere to obtain them, or put their content beyond a paywall.

It’s Changing Rapidly

We’re talking about the Open Web now because of the introduction of AI – a massive disruption to the incentives of many content creators and publishers, because AI both leverages their content (through scraping for training) and competes with it (because it is generative).

For those who opened up their content because they wanted to establish reputation and build connectivity, this feels exploitative. They made their content available to benefit people, and it turns out that it’s benefiting large corporations who claim to be helping humanity but have failed to convince many.

For those who want to sell ads next to their content or entice people to subscribe, this feels like betrayal. Search engines built an ecosystem that benefited publishers and the platforms,but publishers see those same platforms as continually taking more value from the relationship – as seen in efforts to force intermediation like AMP, and now AI, where sites get drastically reduced traffic in exchange for nothing at all.

And so people are blocking bots, putting up paywalls, changing business models, and yanking their content off the Open Web. The commons is suffering because technology (which always makes something easier) now makes content creation and consumption easier, so long as you trust your local AI vendor.

This change is unevenly distributed. There are still people happily publishing open content in formats like RSS, which doesn’t facilitate tracking or targeting, and is wide open to scraping and reuse. That said, there are large swathes of content that are disappearing from the Open Web because it’s no longer viable for the publisher; the balance of incentives for them has changed.

Open is Not Free to Provide

Information may be a non-rivalrous good, but that doesn’t mean it’s free to provide. The people who produce it need to support themselves.

That doesn’t mean that their interests dominate all others, nor that the structures that have evolved are the best (or even a good) way to assure that they can do so; these are topics better suited for copyright discussions (where there is a very long history of such considerations being debated).

Furthermore, on a technical level serving content to anyone who asks for it on a global scale might be a commodity service now – and so very inexpensive to do, in some cases – but it’s not free, and the costs add up at scale. These costs – again, alongside the perceived extractive nature of the relationship – are causing some to block or otherwise try to frustrate these uses.

Underlying this factor is an argument about whether it’s legitimate to say you’re on ‘the Open Web’ while selectively blocking clients you don’t like – either because they’re abusive technically (over-crawling), or because you don’t like what they do with the data. My observation here is that however you feel about it, that practice is now very, very widespread – evidence of great demand on the publisher side. If that capability were taken away, I strongly suspect the net result would be very negative for the Open Web.

It’s About Control

Lurking beneath all of these arguments is a tension between the interests of those who produce and use content. Forgive me for resorting to hyperbole: some content people want pixel-perfect control not only over how their information is presented but how it is used and who uses it, and some open access advocates want all information to be usable for any purpose any time and anywhere.

Either of these outcomes (hyperbole as they are) would be bad for the Open Web.

The challenge, then, is finding the right balance – a Web where content producers have incentives to make their content available in a way that can be reused as much as is reasonable. That balance needs to be stable and sustainable, and take into account shocks like the introduction of AI.

A Way Forward

Having an Open Web available for humanity is not a guaranteed outcome; we may end up in a future where easily available information is greatly diminished or even absent.

With that and all of the observations above in mind, what’s most apparent to me is that we should focus on finding ways to create and strengthen incentives to publish content that’s open (for some definition of open) – understanding that people might have a variety of motivations for doing so. If environmental factors like AI change their incentives, we need to understand why and address the underlying concerns if possible.

In other words, we have to create an Internet where people want to publish content openly – for some definition of “open.” Doing that may challenge the assumptions we’ve made about the Web as well as what we want “open” to be. What’s worked before may no longer create the incentive structure that leads to the greatest amount of content available to the greatest number of people for the greatest number of purposes.