The protocol force

Here’s a riff on the “hourglass model” conversation I’ve had with Bernhard Seefeld a while back. I only recently connected it with my earlier mumblings about software layering, and here’s what came out.

If a technology stack is arranged in such a way that there are consumers of value on one end of the stack and producers of it on the other, then at a certain scale, there emerges a force that encourages one middle layer of the stack – the waist – to be as  thin as possible. There will be a distinct preference for lower diversity of offerings at that layer. IP, HTTP, and HTML are all great examples of this force’s effects.

I am going to call this force the “protocol force”, since the outcome of this force is usually a common protocol or format that is used for communicating across that layer. 

To sketch out the mechanics behind the protocol force, the upper layers want to have access to more consumers, and those are only accessible through the lower layers. The lower layers want to get more producers, and only the upper layers can provide those. To get either, the layers have to go through the waist, and they all want to minimize the cost of bridging, of implementing all the permutations that enable consumers and producers to interact with each other. Put differently, everyone wants to spend as little as possible to reach as many consumers/producers as possible.

For instance, if we have multiple interaction protocols available in the waist layer, a growth spurt of an ecosystem of producers and consumers around this layer will trigger the  “winner takes all” dynamics: a protocol that gets some critical mass of consumers (or producers) will reduce the appeal of using other protocols. As these other protocols will fall into disuse, the one that “won” will continue to gain new adopters  – thus thinning the waist layer.

Interestingly, this protocol force may not manifest in the adjacent layers, even if they look like the connectors between the upper and the lower layers. As long as there’s one layer that is thin, the adjacent layers may enjoy high diversity of offerings. 

A good example of this is the Web frameworks. There are always so many of them, right? Why is that? My explanation of this phenomenon, applying the reasoning above, is that they are protected from the effect of the protocol force by the Web Platform layer (the HTML/CSS/JS combo that we all love and enjoy) that sits right underneath them. This layer is incredibly thin: thanks to the set of circumstances and sheer designers’ luck, we only have one Web Platform.

Because of that, there can be a cornucopia of Web frameworks: as long as what they output is Web Platform-compatible, they won’t experience the pressure to agglomerate or race to the bottom.

The protocol force will be present in any layered stack that has producers and consumers as outer layers. One of the bridging layers will succumb to it. If we’re designing a layered system, we are better off preparing for this force to tap us on the shoulder and ask us to pick a layer to squish. Any resistance is futile. Might as well plan for it.

One of my favorite recent examples of such a design is the Protocol Buffers, a framework that Google uses internally to connect disparate pieces of functionality into one coherent whole. As long as a program speaks protos, it can mingle in Google-land. Whether intentionally or not, protocol buffers serve as the waist layer of Google infrastructure.

One of the key challenges of designing something as versatile as the Web Platform or protocol buffers is the fact that this layer will be the most used and, sadly. abused part of our stack. Once squished into a thin waist, this layer becomes the great enabler of our future – or the source of our doom. Its flexibility and capacity to convey the intentions of developers will be constantly tested and stretched and often broken. This is a sign of success and a good thing. As long as we’re prepared to invest into continuously improving and developing the waist of our stack, we can harness the protocol force to help us, rather than hinder.