In this Halloween-themed episode, I wanted to share a story that might be useful to API engineers, both aspiring and experienced. This is the story of four curses of API development and the shadows that conjure them. So grab that pumpkin spice latte and get ready to hear the 🎃 SPOOOOOKY! 👻 tale.
🕳️ Shadows
First – the shadows. Every upside has a downside. We don’t like to think of them on the upswing. When the shadows visit us, we are unhappily surprised to learn of their existence. This happens so often, we’d think we would have learned by now.
Take the “✨Interest” condition from my earlier story about growing dandelions. Interest is great, right? Unfortunately, with interest and excitement about an API’s potential value comes the shadow of … well, people actually trying to realize this potential value – and extract as much of it as possible.
With the initial spirit of exploration comes the thrust of exploiting: trying to use and – unfortunately, all too commonly – abuse the API to make it do their bidding. If anything, the sudden rise of interest in an API is a warning sign for their vendor: time to think about confronting the shadow of grift that will inevitably emerge.
It is often an uncomfortable job to be the one pointing out the shadow when the team’s collective eyes are on the shiny light of success. Yet, knowing of the existence and anticipating the emergence of the shadow can really save the organization’s hide by helping it orient toward the challenge, rather than be blindsided by it.
However imperfect and goofy, I hope that this narrative will help you do just that. I organized it around eight shadows – one for each condition for growing dandelions and caring for elephants. Think of these eight as the tripwires, the emergent downsides of having been successful at attaining each condition. But to come together, the narrative needs one more twist: the curses.
🧙 Curses
Curses are menacingly sticky. They are imposed on us. No matter how much we try, curses hold us. We can point at them, battle them, and even occasionally proclaim victory over them. But sooner or later, we recognize with a sinking feeling that the celebration was premature. Our curses find yet another way to rear their ugly heads. All we can do is cherish the gift that usually comes with the curse.
The particular kind of curse I want to highlight here emerges from a seemingly innocent concept of idea pace layers. I touched on it briefly in my first article about dandelions and elephants. Ideas thrive as light, free-floating dandelions. Some survive the descent through the ideation pace layers. These ideas grow and create value around them – that is the gift of this descent. As they grow, the conditions of supporting and nurturing them transform to accommodate their growth – to treat them more elephant-like. Somewhere alongside that transformation, the conditions cross the threshold where preserving the accumulated value means more than contemplating change.
Therein hides the curse. Though they still have their strengths and amazing survival abilities within their particular niche, idea-elephants are unable to challenge their shadows. At the bottom of the idea gravity well, we can only make our idea-elephant more precisely formulated and incrementally improve it within its niche – the local maximum.
To find a different local maximum, we need another cycle of exploration: a gazillion of idea-dandelions spreading all over the space, perishing en masse while uncovering precious few novel insights. But to get there, we need conditions that would enable such a development. And such changing of conditions is a threatening proposition when we’re caring for an idea-elephant: starting all over means potentially losing the value we hold. Thus cursed, we flail and struggle to change, but as a rule – fail to do so. The new idea-dandelions can’t find fertile ground in elephant-caring conditions, which makes finding our grip on the elephant shadow even harder.
Even if we’re somehow able to transform ourselves again and recreate favorable conditions for dandelions – it’s not like we’ve gotten away from the curse. As the end credits start rolling, the viewers see our faces being struck by the recognition that we’re starting the cycle all over again.
In the API developer’s world, the progression of this curse can be described as a cadence of steps. It begins with a success, when the conditions we’ve created for dandelion APIs actually start bearing fruit. There are lots of consumers of the APIs and they are starting to build eye-popping things. Somewhere around here, the dandelion shadow is discovered, and we valiantly face the challenges it presents. Whether we know it or not, this process transforms our requirements to create idea-elephant conditions. In the moment, it always makes sense — now that there are successful businesses running on our APIs, this feels like a logical next step. As we do so, the elephant shadow manifests, and forces us to recognize that we need to get back to conditions that are more dandelion-like – and, despite our efforts, the curse prevents us from doing so.
Pairing the four conditions (one from the dandelion-growing list, one from the elephant-keeping one), we end up with four such progressions, the four curses. I’ll call them, respectively, the curse of irrelevance, the curse of immensity, the curse of immobility, and the curse of inscrutability.
🏚️ The Curse of Irrelevance
The two polar conditions in this curse are “ ✨ Interest” for dandelions and “ ⚓️ Stability” for elephants. I already described the moment of discovering the first shadow. I’ve lived that moment a bunch of times throughout my career, and it’s almost always followed by the call to bring things under control. This exertion of control is transformational: it brings the change of conditions toward Stability.
Once that change is complete, we enter the third beat of the curse: encountering the shadow of Stability. It turns out, once we’ve gotten things under control, these things get boring and stale. That same explosive growth, attenuated by the faucet of predictability, slows down to a trickle.
Facing this second shadow, we try to bring back the mojo – and more than likely, can’t. Idea-elephants don’t travel upward in the pace layers. No matter how much we try, new ideas are quickly shot down: too risky, too crazy, too irresponsible. The hard-earned stability resists being disturbed, cursing us with irrelevance.
♾️ The Curse of Immensity
The second curse can be seen as the interplay between “🔮 Legibility” and “⛰ Breadth”. The gift of legibility is in the simplicity with which the API can be used. It’s just begging us to play with it.
However, once our customers start messing with the API, something interesting happens: they start seeing the edges of our canvas, bumping into the limits: “Oh, I wish this API supported this <feature>!” As the dandelion idea of an API takes root in the collective minds of its consumers, there’s a steady stream of requests for improvements. Obliging to fulfill these requests is the second beat of the curse – the transformation to Breadth.
On cue, the shadow of Breadth presents itself: the bloated, incoherent, everything-bagel API surface. Adding new features to the API is a puzzle with many moving pieces. Removing APIs is a massive pain in the butt. Everything around us is gigantic – the scale of our usage, the number of feature requests that keep showing up. And of course – the rising chorus of complaints that the API surface is just too darned large.
Steeling ourselves to confront the second shadow, we discover that it’s much harder to tame than the first one. A common API designer’s trope that I’ve seen (and tried to use myself) is the “well-lit paths” pattern. It seems logical that if we just highlighted some APIs and not others and organized them into well-designed pathways for developers, then some of our incoherence issues would go away. I’ve yet to see a great application of this pattern. Instead, what typically happens is something of a high-modernist paving of lonely highways and bridges to nowhere that adds to the confusion and girth rather than alleviating it. Mocking us, the curse of immensity knows that organizing large API surfaces only makes them larger.
I’ve already written a bit about API deprecation. Deprecation of APIs tends to be a losing battle. It takes a lot more time and effort to remove features than to add them, which means that over time, the tyranny of the curse of immensity only strengthens.
🧊 The Curse of Immobility
Between the conditions of “🚀 Velocity” and “📚 Rigor”, we find the third curse. A setting that allows us to string together a quick prototype is rarely the same setting that we use for launch. As soon as our API customers start seeing some uptick in their usage, the first shadow will immediately remind us of that lesson.
As a matter of transformation, we overcome this shadow by introducing processes and infrastructure that are critical for shipping products at scale. If we are to retain our customers and set them up for long-term success, we must transition to the stance of Rigor.
Pretty soon, the shadow of Rigor makes itself known. All these amazing best practices, checks and balances, launch gates and test infrastructure reduce velocity, sometimes quite dramatically. Gone are the days when one could quickly put together a bug fix. Everything seems to take eons to get done.
This one is especially hard for engineers. Everyone seemingly notices this, yet there does not appear to be a way out. Another rallying cry to make things go faster gets mired in yet another committee or working group. Once the API conditions transform into caring for elephants, getting it back to the lightweight experimentation is prevented by the curse of immobility.
🗝️ The Curse of Inscrutability
The final curse is formed by pairing of “🔎 Access” and “⚡️ Power”. The key tension here is in the level of opinion within the API. Access needs APIs to be highly opinionated, while Power needs the opposite.
The first shadow becomes visible when our users start using the APIs in earnest, beyond initial prototypes. All that opinion that made it possible for them to build those prototypes quickly starts getting in the way. “That’s so cool! How do I turn it off?” was one of my favorite bits of developer feedback to some of my early Web Components API ideas. As developers’ ideas start holding value, the focus shifts to getting closer to the metal.
One of the common drivers in this transformation to the condition of Power happens as a result of trying to squeeze a bit more performance or capabilities out of the product, built on top of the API. This story typically involves the API vendor exposing deeper and deeper hooks inside, and thus relinquishing some (or all) of the opinion held by these APIs. A while back, I already mentioned the Canvas API in WebKit, which collapsed the whole of the HTML/CSS opinion straight to Apple’s GCContext API, which was as close to the underlying platform one could get back then.
As predictably as a Greek tragedy’s plot, the second shadow makes its entrance. With power comes the need for skill to wield this power, which in turn leads to rapid decline in the number of folks who can actually use it effectively. In such scenarios, there are only a few (grumpy) wizards who actually know how to use the APIs, and whoever hires them accrues all the value.
And of course, it is very, very hard to argue convincingly that this value needs to be lost and the power given up to return to the Access condition to confront the second shadow. The curse of inscrutability has taken its hold.
🧛 Haunted API design
The four curses accost us simultaneously and often interplay with each other, usually to a reinforcing effect. The curse of Immensity invites Inscrutability. The curse of Immobility often comes on the heels of those two. The curse of Irrelevance stokes the fears of obsolescence and exacerbates the effect of the other curses. It’s all a hauntingly accursed mess. There is seemingly no escape from it. At least based on my experience, every team that sets out to ship API comes under the spell of these curses.
What’s an API designer to do? Clearly, scream and wail in horror – what kind of Halloween tale would it be otherwise? Oh well. Perhaps some future episode will point the path out of the spine-chilling quagmire. Maybe in time for Christmas?