Dandelions and Elephants

  • r/K selection and innovation

    I’ve been thinking about the different conditions under which innovation emerges, and how the environment influences the kind of innovation that happens.

    To set the stage, I am going to do a very brief detour into biology (I am not a biologist, so will definitely make a mess of it) and use the r/K-selection lens to guide this story. If we observe various organisms, the theory goes that all fall somewhere in the spectrum between r-selection and K-selection. The r-selection is a strategy that’s focused on reproduction: make copies of my kind as quickly as possible, and mutate generation to generation. The K-selection is a carrying capacity strategy: stay alive as long as possible, learn and teach my (usually very few) offsprings the necessary tricks to thrive. Both are viable strategies, depending on the environment.

    Typical examples used for r-selection are dandelions and for K-selection – elephants. To bring it closer to home, we can have the coronavirus represent the side of the r-selection, and us humans the K-selection. The battle of pandemic/endemic we’re still locked in shows that humans are likely to win, but oh boy is COVID giving us the run for the money.

    Turning to the realm of ideas, it seems that some of them are r-selected. In the most basic form, ideas are memes: easily transmitted, rapidly spreading earworms. Each transmission is an opportunity for mutation. As they inhabit our mind, ideas fall in the fertile ground of other ideas and the next time we communicate these ideas, we – nearly always – change them. They mutate into something different. As the r-selected strategy suggests, ideas evolve as they are copied.

    Other kinds of ideas are K-selected. These are usually larger, self-coherent entanglements of smaller ideas. They live in our minds and they evolve over time, growing and maturing. Just like K-selected organisms, these ideas reproduce with pain and effort. We can’t just give them to others. We must garner our patience and engage with them in the process of teaching, smuggling them across the often unreliable medium of interpersonal communication, one tiny elemental bit at a time.

    My fellow FLUX-er Erika pointed out that by putting these two strategies into a spectrum, a framing of pace layers pops out. There are simple ideas boiling afroth at the top layer in a pure r-selected strategy. As they connect with other ideas, their strategy becomes more and more K-selection-like, forming layers that move at a slower pace. Learning Newtonian physics? Welcome to the lower layers. Reading the latest memes on Twitter? That’s at the top.

    This pace layering offers us a neat way to look at the kinds of innovation. Since innovation is powered by ideas, we can see it manifest in different ways at different pace layers. The higher, more r-selected layers are great for rapidly exploring new spaces, where there are lots of unknowns and possibilities. The lower, more K-selected layers are incredibly effective for optimizing and refining existing ideas. As ideas mature, they traverse the pace layers, descending from r to K.

    Using this lens, an organization that desires to innovate can act more intentionally by picking the idea selection strategy. If we’re looking at a new wide-open space, or searching for the next frontier, we would want to create conditions where ideas can spread rapidly and replicate to mutate. 

    A recent example of this happening is the release of Stable Diffusion, an AI model for generating images from prompts. Simon Willison has an insightful write-up of the phenomenon, and we don’t have to squint to spot application of the r-selected strategy: the model and the surrounding tools are open source, and the process of setting up your own instance is fairly straightforward. 

    Combined with rising interest around the future potential of generative media, this caused an immediate explosion of innovation: people messing with it for fun and/or quickly putting together possible business ideas. Given that no other contenders offered such an opportunity to the interested crowd, I would not at all be surprised if Stable Diffusion, no matter how well it fares in its comparative quality today, becomes the de-facto engine for generating pictures from text.

    On the other hand, if we want to climb the gradient hill and improve upon an existing idea, we are better off picking the K-selected strategy: invest into apprenticeship practices to facilitate tacit knowledge transfer, and ensure that idea refinements are carefully curated. 

    I will once again lean onto browser development to illustrate the K-selected strategy. The modern Web rendering engine, which is the thing that interprets text, images, and code as a visual, interactive composition on your screen that we call “Web sites” is a prototypical outcome of descending down the pace layers.

    A wild-haired idea at first, it eventually grew into a massively complicated and often self-contradictory tangle of ideas that is captured in code. There are only three distinct instances of that code: WebKit, Blink, and Gecko. These instances try to interoperate, and there is an incredible amount of knowledge — both tacit and described in long, dry documents called specs  – that need to be grokked before proceeding with constructing another instance. If Stable Diffusion is a dandelion, Web rendering engines are elephants. 

    One does not simply create a new rendering engine. To move forward, a K-selected strategy is called for. When we look at the efforts surrounding creating a rendering engine, we can see extensive documentation and education that helps aspiring Web browser engineers climb the learning curve, as well as rigorous processes to ensure that every change that is being made to the codebase carefully, yet purposefully evolves the project.

    Similarly, we can anticipate mismatches in the kind of innovation we’d like to see and the strategy that we have at our disposal. We can’t expect to explore a new space with data-driven analyses and an elite team of experts — just like we can’t expect to see consistent results aimed toward some goal if we’re set up for rapid, unbridled experimentation.

  • Growing dandelions

    A couple of weeks ago, I talked about r/K-selection and mentioned two kinds of ideas: the mutate-through-replication dandelions and capacity-preserving elephants.

    I remain curious about the conditions in which dandelions thrive. Here are some initial thoughts on the subject. To narrow the broad designation of “ideas” a bit, I am going to focus on a special case:  innovation on top of APIs. That is, new ideas that emerge while writing code that consumes some set of APIs.

    Let’s suppose we’re just starting down the process of designing a new API. Very early, we decided that we want this API to spur dandelions. We did a lot of thinking and realized that our fledgling enterprise will benefit greatly from employing the r-selected strategy.

    Why would we want to do that? Primarily, the r-selected strategy works best in environments that aren’t (yet) predictable or stable, where the rate of change is high. For example, we might be entering some new problem space and we want to lean onto the “wisdom of the crowd” to explore it. Or perhaps we’re a newcomer and we would like to convert the budding enthusiasm in the problem space into as many dependencies on our services as possible (if you’re looking for a case study on both, check out Stable Diffusion playbook).

    What are the conditions that we need to grow dandelions? How might we design APIs that encourage dandelion-like innovation?

    Using the r/K-selection in biology as our guide, we see four key conditions: high reproduction rate, small size, short generation time, and wide dispersion radius. Yes, just like dandelions.

    Translating these conditions from plants to ideas, I came up with: interest, legibility, velocity, and access. Let’s go through them one by one. You know me. I love my “let’s go through them one by one” bit.

    ✨ Interest

    First, this API actually needs to promise to unlock exciting new opportunities. This leads me to the first condition: interest, which roughly matches the “high reproduction rate” in biology. Exciting  ideas are contagious. They spur lots of new ideas, churning them out at a high rate. They don’t even have to have concrete value behind them – just a promise of something big and potentially groundbreaking. This sometimes leads to formation of a bubble of hype around them, like with Web3 and the NFTs. Such bubbles, while not healthy in the long term, are a strong sign of the interest condition being met.

    Interest is not an intrinsic property of the API design, but rather the property of the technology behind it. Researchers teased the developer community with the tantalizing potential of AI-generated media for years now, building up the interest in the underlying technology. It was OpenAI and then Stable Diffusion who capitalized on this interest and shipped first publicly-accessible APIs that enabled developers to actually play with technology. The resulting wave of innovation was nothing short of astounding – and it keeps going. There ought to be a clock of “days since Stable Diffusion was released” somewhere, because the quantity of interesting new ideas born out of that event feels unbelievable when put in the context of the little time that had passed. Again, a great example of the interest condition being met.

    To give you a counter-example, consider OpenSocial,  a cool idea from way back in 2007 that started with much fanfare at Google and ended up dying quietly in the W3C effort graveyard. Even though yours truly did end up playing with it, very few others did. Was it ahead of its time? Was it too obtuse? Was it the XML thing? We will never know. But if your API adoption patterns are looking like those of OpenSocial, check the pulse of the community interest.

    🔮 Legibility

    Another significant condition is legibility. To allow an idea to spark imagination and produce new ideas, it must be easily understood – even if partially. I correlated this one with the biological counterpart of “small size”. Metaphorically, think of it this way: an idea that is light and small like a dandelion is much easier to grasp than the weighty idea-elephant. A litmus test: can a useful program built with our API fit into a tweet? The deca-LOC framing is useful here, though it’s not just the number of lines of code. 

    One of the key tenets of the WebKit open source project at one time was the idea of self-explanatory code: is your change making the code more or less easy to read? There were some who even suggested that adding comments is somewhat of a code smell: if you have to explain what it does, then perhaps maybe you could write it more eloquently instead? 

    While I don’t take this extreme point of view, I appreciate the sentiment. Choosing the idioms and concepts that make the code concise while capturing the key thrust of the idea behind the API is difficult work and is full of difficult trade-offs. Sometimes it takes several iterations to arrive at a mental model of the API that clicks with most people. When designing for dandelions, opinion is front and center, and the easiest-to-grasp concepts win over the more obtuse ones – even if the latter are more powerful and flexible.

    Speaking of WebKit, and similar large codebases. A key ingredient of legibility is the ease with which an idea can be separated from other ideas. How discrete is it? How easy is it to spot this idea and lift it out? WebKit has a ton of great ideas in its code. I know, I lived in that repository a few years back, and I bet there are even more flashes of brilliance now. However, to spot them as separate ideas, we have to spend a bunch of time understanding how all the surrounding neighboring ideas fit.

    This is one of the challenges of implementing r-selected strategies in large code repositories. Now matter how we try, our dandelions end up being somewhat elephant-like.

    🚀 Velocity

    The third condition has to do with how quickly the ideas borne out of interacting with our APIs can turn into other, new ideas. This is not necessarily a property of the API itself, but rather a setting into which it is born. Loosely, it corresponds to “short generation time” in biology. Velocity is commonly called “tight developer feedback loop” in developer experience jargon, and yes, it’s that, and a bit more. I see it as somewhat two-fold: time to result and effort to copy.

    Time to result is the time it takes between making a change to the code and seeing the results of the change. Back when I first started using computers, I remember working with a particularly old piece of equipment that provided the output of my program only as a paper printout – and the printer was across the hall from the monitor and keyboard. Time to result included jogging out of the lab and into the computer room, where massive printers hammered out loudly our many failures and rare successes. Paper jam? Well, you might have to run that job again. Even just capturing the simplest idea into working code was a multi-hour (and sometimes, multi-day) process. That mini-computer was from the pre-dandelion era. The shorter the time to result, the better velocity of an idea.

    Effort to copy is an adjacent concept. How much effort does it take to copy an idea? Is it a complicated process? Or is it just one click? A somewhat unexpected, yet obvious-upon-inspection factor here is organizational boundaries. 

    Back when I worked on Google Gears as an external-to-Google contributor, I was puzzled at the weird phenomenon: to land, my patches would need to be sent over email as diff files. The open-source directory did not contain any commits from individuals: instead, every commit was made by a bot. After a couple of days of submitting the patch, the bot would dutifully add my commit to the repository. What the heck was happening?! As one of the engineers explained, the actual source of truth was on the other side of the wall that separated the inside of Google from the outside. To land the code, a Google engineer had to patch it in, have it reviewed, and then let the bot take it outside. What I was working with was a mirror, not a real thing. Sure, the automated bot made things easier. But across the wall like this, the effort to copy is still high – and not just for the code going out. If there is some really cool new innovation on the outside of the wall, an organization has little choice but to rebuild it – often from scratch – on the inside. 

    On the other side of the spectrum, Github’s “fork” button is a great example of intentionally lowering effort to copy. Want to play with an idea? Click and start making it yours. As another illustration, both effort to copy and time to result are combined delightfully into various read-eval-print loop (REPL) tools that sprouted all over the place in the past decade. Though my first love was JS Bin (hi Remy!), one of my favorite ones today is Replit, which seems to be designed by someone who deeply understands the concept of dandelion gardening.

    🔎 Access

    The final condition is access. It seems that, to stimulate r-selection, we need a large pool of minds that our APIs can come in contact with. To generate many ideas, we need many minds – or to have a “high dispersion radius”, speaking biologically.

    What does it take to start using our API? What are the barriers that the person must overcome? For example, if we decide to provide our API in some programming language that nobody ever heard of, we are increasing the barrier. To access our APIs, people first have to learn this language. 

    As perhaps a somewhat controversial example, in the very early days of Flutter, we had this contentious debate whether the engine would rely on Javascript or Dart. Though Dart won in the end, I wonder how much more widespread the use of Flutter would have been had we stayed with Javascript. In other words, is Flutter successful because of or despite Dart?

    Dandelion-growing needs space. If we’re planning a small group within our organization as the potential users of the API, we are unlikely to get any benefits of the r-selected strategy. This is often counterintuitive in organizations that pride themselves on engineering excellence. It feels like we should be able to just get a couple of really smart folks to play with the API, and they will figure out some interesting possibilities… right? Well, maybe. 

    But if we are aiming to harness the r-selected strategy, we need to stop looking for experts who might give us great insights. Instead, we need to open the API up as broadly as possible and let the wave of hobbyists and enthusiasts wash over it. When growing dandelions, think quantity over quality. Skill and expertise are a barrier.

    Additionally, to maximize the number of ideas connecting with each other, I need to make them easy to find and browse. Can a seed of an idea be easily discovered? Can I trace its heritage and find earlier seeds on which the idea was based? Can I see who else is playing with it currently? And no less importantly, who can find my idea?

    A gardener’s guide

    Putting these all together, we can build a simple compass. The four conditions form an arrow that points us toward dandelion-like growth:

    • Is the underlying technology that the API exposes interesting? Do we anticipate developer buzz around it?
    • Is the API mental model easy to grasp? Can developers’ ideas be expressed in simple, elegant code? Is it free of dependencies that developers might be unfamiliar with?
    • Is the setting into which we release the API offer REPL-like iteration speed and one-click copying of ideas?
    • How large is the pool of people who could conceivably use the API? Is the cost to entry minimal? Is the list of prerequisite learnings short? Is it easy to find similar or different ideas and understand how they came together?

    There are a handful of folks that I know who seem to intuitively understand these conditions, and their approaches to API development reflects that. For the rest of us – myself included – here’s hoping that this compass will serve us well in our dandelion-growing pursuits.

  • Caring for elephants

    Now that we have a guiding compass for growing dandelions, what of the elephants? What are the conditions that might be effective for our APIs to nurture ideas that are like elephants? 

    As a quick reminder, elephants, as all species that rely on K-selected strategy in biology, are characterized by these four characteristics that are roughly the opposite of dandelions (the r-selected strategy).

    First, constant size of the population is important in an environment at or near its carrying capacity, so K-selected strategy encourages low reproduction rate.

    Second, to survive through ebbs and flows of resources within the particular ecological niche, an organism needs mass. K-selected bodies are usually larger in their size.

    Third, rather than let mutation take care of finding the fit within their niche, there’s an additional band of adaptation – through knowledge. K-selected species learn and change their behavior over their lifetime. This encourages a longer life span and a longer maturation process. Children take a while to become adults and grown-ups invest time passing their learning on to their offsprings.

    Fourth and final characteristic is a set of particular strengths. These come handy when K-selected organisms compete for limited resources in a crowded niche. Be it flexibility, agility, or just plain brawn – like for elephants, each is carefully selected for over a long curve of evolutionary selection.

    Before we go any further, a reasonable question: why would one anyone want to build APIs for idea-elephants? The motivation is usually somewhere around their size. Idea-elephants tend to hold and retain value. If we want to build APIs to help us generate reliable revenue for a long period of time, we are probably looking for an elephant-caring strategy.

    A good example of an idea-elephant is an ecosystem: people and technology mingling together in mutually beneficial ways. Thriving ecosystems have lasting power and behind each, there’s a learned way of doing things, the idea that defines the nature of the ecosystem. Unlike with dandelions, the idea is no longer freely mutable. The Internet is one of those gigantic elephants, with the Internet Protocol as the API that makes it possible.

    At a smaller scale, anytime we want to preserve some value that we believe is contained in the use of a particular technology, we’ll likely want to create favorable conditions for elephants in our API design.

    Just like in the previous exercise with dandelions, I’ll map the biological attributes to equivalent conditions.

    ⚓️ Stability

    Unlike dandelions’ obsession with excitement, elephants tend to want their dependencies to be boring. Being predictable and reliable is a highly sought-after quality.  Earned trust is of the essence for the APIs that aspire to cater to elephants. Trustworthy API design is somewhat of an art form. In some ways, the seemingly easiest move is not to change anything. Every change might result in a potential breakage and loss of held value. Holding that value is more important than pursuing new ideas, and as such, the number of new ideas (the “reproduction rate” from biology) will be small.

    One of the projects I helped start at the Chrome Web Platform team was the predictability effort, to identify and address key gotchas and inconsistencies that frustrate Web developers. The strategic thrust of this effort was to help make the Web Platform APIs more hospitable to the elephant of the Web developer ecosystem.

    Every organization that accumulates value in its own infrastructure and code usually ends up investing in making both as stable and reliable as possible. An entire profession of Site Reliability Engineers (SREs) emerged to represent the special skill that’s required in making that happen. 

    Breadth

    Elephant-tending APIs cover a lot of ground. They are large (the biological “large size” equivalent). There are lots of use cases that accumulate over time and each unaddressed use case is a missed opportunity, a value loss. A very common thing that happens with dandelion APIs that become popular is that they grow in size and complexity. When that happens, we are observing a relatively rare event: the API is moving down through the pace layers, becoming more and more elephant-like.

    When Javascript was first introduced to the Web, the API to access the document tree (aka Document Object Model) was exceedingly simple. Just a few objects, a simple way to access things, and that’s about it – poorly documented, a “figure it out” dandelion spirit. 

    Today’s DOM API is quite large, and it doesn’t even include most of the Web Platform APIs – to keep the DOM spec light, a notion of partial interfaces was introduced. The HTML spec captures the bulk of the vast surface. Give your scrolling muscles a go to get a sense of the breadth.

    This is a fairly common occurrence when caring for idea-elephants with our APIs: their needs are rarely captured in a few simple calls.

    📚 Rigor

    Just like with dandelions, some conditions reflect the setting, rather than the APIs themselves. This is the case for rigor. As ideas expand to become elephants, they trade their agility for rigor as a strategy to extend lifetime (“longer lifespan” in biology). Haunted by the danger of potential lost value, decisions are made more carefully and thoughtfully. 

    One does not simply push to production in elephant-land. There are feature launch calendars, approval gates, and deliberate release processes that move developers through. New ideas emerge very slowly – and for good reason. Ideas must be tested to fit well with the massive body of existing ideas. Reducing uncertainty triumphs over explosive innovation.

    For example, shipping a new Web platform feature in Blink (the Web rendering engine in Chromium) is a six-step process that involves building out an initial set of use cases, potentially asking for a mentor to help with specification writing, making a proposal in a standards organization, and socializing the problem with other browser vendors and Web developers. And by the way, all these items are just part of step one.

    The upside of such a slow process is that the ideas that do make it through are at full maturity. Like elephant calves, they have to be slowly nurtured and “taught” all of the intricacies of the wisdom that affords the massive scale and value of an elephant.

    ⚡️ Power

    The last, but definitely not least condition is power, which neatly matches the biological equivalent of a set of strengths. Idea-elephants gravitate toward — and often demand — more powerful and less opinionated APIs. Put simply, elephants want to be closer to the metal. Unlike dandelions, elephants have the capacity to hold their own opinions. In fact, some of these opinions might be load-bearing: the value that an elephant so carefully wants to preserve is based on them. Presenting them with other opinions might appear foolish or downright hostile.

    One of the common struggles when designing APIs for the Web for me was trying to resolve that constant pressure from Web framework developers wanting to see powerful, low-level APIs and the declarative spirit of the Web. If you read this blog, you probably remember my stories about layering: all of them come from this weird paradox that the Web platform houses both elephants and dandelions in the same massive farmhouse.

    When we are designing for idea-elephants, we are much better off letting the elephants hold their opinions, and concentrate on low-level, opinion-less abstractions that delegate most of the power to the elephants we are caring for.

    Caregiver’s Guide

    What can we learn from this exercise? Here is a set of questions we can ask ourselves when looking to care for idea-elephants:

    • Are the APIs we offer predictable, reliable, stable? Can we guarantee providing them for a long period of time and continuously reducing any inconsistencies or bugs that might creep in?
    • Are our APIs comprehensive and cover most of the use cases that the developers are asking for? Do we commit to improving this coverage over time?
    • Does the setting into which we release the API have the necessary infrastructure for ensuring that the API consumers make good choices, from robust integration and testing to deployment processes, as well as telemetry and safe experimentation, locally and in the wild?
    • Do our APIs take developers as close to the underlying technologies as possible, offering little of its own opinion in the process?

    An interesting question to ponder: so far, I’ve been only talking about conditions necessary for taking care of an elephant. I didn’t mention anything about the conditions for creating one, like I did with dandelions.

    My intuition here is that elephants are rarely created from whole cloth. Every elephant-idea begins as a rare dandelion-idea that managed to grow and accrue value over a long period of time, traversing the idea pace layers. So, we rarely choose to be in the position of elephant caregiver: it’s something that happens to us as a result of our idea’s success.

  • Shadows and Curses

    In this Halloween-themed episode, I wanted to share a story that might be useful to API engineers, both aspiring and experienced. This is the story of four curses of API development and the shadows that conjure them. So grab that pumpkin spice latte and get ready to hear the 🎃 SPOOOOOKY! 👻 tale.

    🕳️ Shadows

    First – the shadows. Every upside has a downside. We don’t like to think of them on the upswing. When the shadows visit us, we are unhappily surprised to learn of their existence. This happens so often, we’d think we would have learned by now.

    Take the “✨Interest” condition from my earlier story about growing dandelions. Interest is great, right? Unfortunately, with interest and excitement about an API’s potential value comes the shadow of … well, people actually trying to realize this potential value – and extract as much of it as possible. 

    With the initial spirit of exploration comes the thrust of exploiting: trying to use and – unfortunately, all too commonly – abuse the API to make it do their bidding. If anything, the sudden rise of interest in an API is a warning sign for their vendor: time to think about confronting the shadow of grift that will inevitably emerge.

    It is often an uncomfortable job to be the one pointing out the shadow when the team’s collective eyes are on the shiny light of success. Yet, knowing of the existence and anticipating the emergence of the shadow can really save the organization’s hide by helping it orient toward the challenge, rather than be blindsided by it.

    However imperfect and goofy, I hope that this narrative will help you do just that. I organized it around eight shadows – one for each condition for growing dandelions and caring for elephants. Think of these eight as the tripwires, the emergent downsides of having been successful at attaining each condition. But to come together, the narrative needs one more twist: the curses.

    🧙 Curses

    Curses are menacingly sticky. They are imposed on us. No matter how much we try, curses hold us. We can point at them, battle them, and even occasionally proclaim victory over them. But sooner or later, we recognize with a sinking feeling that the celebration was premature. Our curses find yet another way to rear their ugly heads. All we can do is cherish the gift that usually comes with the curse.

    The particular kind of curse I want to highlight here emerges from a seemingly innocent concept of idea pace layers. I touched on it briefly in my first article about dandelions and elephants. Ideas thrive as light, free-floating dandelions. Some survive the descent through the ideation pace layers. These ideas grow and create value around them – that is the gift of this descent. As they grow, the conditions of supporting and nurturing them transform to accommodate their growth – to treat them more elephant-like. Somewhere alongside that transformation, the conditions cross the threshold where preserving the accumulated value means more than contemplating change. 

    Therein hides the curse. Though they still have their strengths and amazing survival abilities within their particular niche, idea-elephants are unable to challenge their shadows. At the bottom of the idea gravity well, we can only make our idea-elephant more precisely formulated and incrementally improve it within its niche – the local maximum.

    To find a different local maximum, we need another cycle of exploration: a gazillion of idea-dandelions spreading all over the space, perishing en masse while uncovering precious few novel insights. But to get there, we need conditions that would enable such a development. And such changing of conditions is a threatening proposition when we’re caring for an idea-elephant: starting all over means potentially losing the value we hold. Thus cursed, we flail and struggle to change, but as a rule – fail to do so. The new idea-dandelions can’t find fertile ground in elephant-caring conditions, which makes finding our grip on the elephant shadow even harder.

    Even if we’re somehow able to transform ourselves again and recreate favorable conditions for dandelions – it’s not like we’ve gotten away from the curse. As the end credits start rolling, the viewers see our faces being struck by the recognition that we’re starting the cycle all over again. 

    In the API developer’s world, the progression of this curse can be described as a cadence of steps. It begins with a success, when the conditions we’ve created for dandelion APIs actually start bearing fruit. There are lots of consumers of the APIs and they are starting to build eye-popping things. Somewhere around here, the dandelion shadow is discovered, and we valiantly face the challenges it presents. Whether we know it or not, this process transforms our requirements to create idea-elephant conditions. In the moment, it always makes sense — now that there are successful businesses running on our APIs, this feels like a logical next step. As we do so, the elephant shadow manifests, and forces us to recognize that we need to get back to conditions that are more dandelion-like – and, despite our efforts, the curse prevents us from doing so. 

    Pairing the four conditions (one from the dandelion-growing list, one from the elephant-keeping one), we end up with four such progressions, the four curses. I’ll call them, respectively, the curse of irrelevance, the curse of immensity, the curse of immobility, and the curse of inscrutability.

    🏚️ The Curse of Irrelevance

    The two polar conditions in this curse are “ ✨ Interest” for dandelions and “ ⚓️ Stability” for elephants. I already described the moment of discovering the first shadow. I’ve lived that moment a bunch of times throughout my career, and it’s almost always followed by the call to bring things under control. This exertion of control is transformational: it brings the change of conditions toward Stability. 

    Once that change is complete, we enter the third beat of the curse: encountering the shadow of Stability. It turns out, once we’ve gotten things under control, these things get boring and stale. That same explosive growth, attenuated by the faucet of predictability, slows down to a trickle. 

    Facing this second shadow, we try to bring back the mojo – and more than likely, can’t. Idea-elephants don’t travel upward in the pace layers. No matter how much we try, new ideas are quickly shot down: too risky, too crazy, too irresponsible. The hard-earned stability resists being disturbed, cursing us with irrelevance. 

    ♾️ The Curse of Immensity

    The second curse can be seen as the interplay between “🔮 Legibility” and “⛰ Breadth”. The gift of legibility is in the simplicity with which the API can be used. It’s just begging us to play with it. 

    However, once our customers start messing with the API, something interesting happens: they start seeing the edges of our canvas, bumping into the limits: “Oh, I wish this API supported this !” As the dandelion idea of an API takes root in the collective minds of its consumers, there’s a steady stream of requests for improvements. Obliging to fulfill these requests is the second beat of the curse – the transformation to Breadth. 

    On cue, the shadow of Breadth presents itself: the bloated, incoherent, everything-bagel API surface. Adding new features to the API is a puzzle with many moving pieces. Removing APIs is a massive pain in the butt. Everything around us is gigantic – the scale of our usage, the number of feature requests that keep showing up. And of course – the rising chorus of complaints that the API surface is just too darned large.

    Steeling ourselves to confront the second shadow, we discover that it’s much harder to tame than the first one. A common API designer’s trope that I’ve seen (and tried to use myself) is the “well-lit paths” pattern. It seems logical that if we just highlighted some APIs and not others and organized them into well-designed pathways for developers, then some of our incoherence issues would go away. I’ve yet to see a great application of this pattern. Instead, what typically happens is something of a high-modernist paving of lonely highways and bridges to nowhere that adds to the confusion and girth rather than alleviating it. Mocking us, the curse of immensity knows that organizing large API surfaces only makes them larger.

    I’ve already written a bit about API deprecation. Deprecation of APIs tends to be a losing battle. It takes a lot more time and effort to remove features than to add them, which means that over time, the tyranny of the curse of immensity only strengthens.

    🧊 The Curse of Immobility

    Between the conditions of “🚀 Velocity” and “📚 Rigor”, we find the third curse. A setting that allows us to string together a quick prototype is rarely the same setting that we use for launch. As soon as our API customers start seeing some uptick in their usage, the first shadow will immediately remind us of that lesson.

    As a matter of transformation, we overcome this shadow by introducing processes and infrastructure that are critical for shipping products at scale. If we are to retain our customers and set them up for long-term success, we must transition to the stance of Rigor.

    Pretty soon, the shadow of Rigor makes itself known. All these amazing best practices, checks and balances, launch gates and test infrastructure reduce velocity, sometimes quite dramatically. Gone are the days when one could quickly put together a bug fix. Everything seems to take eons to get done.

    This one is especially hard for engineers. Everyone seemingly notices this, yet there does not appear to be a way out. Another rallying cry to make things go faster gets mired in yet another committee or working group. Once the API conditions transform into caring for elephants, getting it back to the lightweight experimentation is prevented by the curse of immobility.

    🗝️ The Curse of Inscrutability

    The final curse is formed by pairing of “🔎 Access” and “⚡️ Power”. The key tension here is in the level of opinion within the API. Access needs APIs to be highly opinionated, while Power needs the opposite. 

    The first shadow becomes visible when our users start using the APIs in earnest, beyond initial prototypes. All that opinion that made it possible for them to build those prototypes quickly starts getting in the way. “That’s so cool! How do I turn it off?” was one of my favorite bits of developer feedback to some of my early Web Components API ideas. As developers’ ideas start holding value, the focus shifts to getting closer to the metal.

    One of the common drivers in this transformation to the condition of Power happens as a result of trying to  squeeze a bit more performance or capabilities out of the product, built on top of the API. This story typically involves the API vendor exposing deeper and deeper hooks inside, and thus relinquishing some (or all) of the opinion held by these APIs. A while back, I already mentioned the Canvas API in WebKit, which collapsed the whole of the HTML/CSS opinion straight to Apple’s GCContext API, which was as close to the underlying platform one could get back then.

    As predictably as a Greek tragedy’s plot, the second shadow makes its entrance. With power comes the need for skill to wield this power, which in turn leads to rapid decline in the number of folks who can actually use it effectively. In such scenarios, there are only a few (grumpy) wizards who actually know how to use the APIs, and whoever hires them accrues all the value. 

    And of course, it is very, very hard to argue convincingly that this value needs to be lost and the power given up to return to the Access condition to confront the second shadow. The curse of inscrutability has taken its hold.

    🧛 Haunted API design

    The four curses accost us simultaneously and often interplay with each other, usually to a reinforcing effect. The curse of Immensity invites Inscrutability.  The curse of Immobility often comes on the heels of those two. The curse of Irrelevance stokes the fears of obsolescence and exacerbates the effect of the other curses. It’s all a hauntingly accursed mess. There is seemingly no escape from it. At least based on my experience, every team that sets out to ship API comes under the spell of these curses.

    What’s an API designer to do? Clearly, scream and wail in horror – what kind of Halloween tale would it be otherwise? Oh well. Perhaps some future episode will point the path out of the spine-chilling quagmire. Maybe in time for Christmas? 

  • Dandelion or Elephant?

    The discovery of the dandelion/elephant framing was exciting and my fellow FLUX colleagues and I engaged in a rather fun “hacky sack of ideas” game, tossing the framing back and forth and looking at it from this side and that. One pattern that emerged was the “dandelion/elephant” test: is this company/team/product/concept a dandelion or an elephant? The test kept producing unsatisfactory results, making us wonder: are we holding this wrong? As usual, some new insights emerged. I will try to capture them here.

    First things first: it is very easy to get disoriented about what it is that we’re testing. In our excitement, we’d forgotten that the biological equivalents of our subjects are strategies. The r-selected strategy and the K-selected strategy are approaches to the problem space that various species take. Similarly, “dandelion” or “elephant” aren’t attributes or states of an organization or product. They are strategies that an entity chooses to overcome a challenge it faces. In other words, it’s not something that an entity is or has, but rather how it acts.

    Since it deals with strategies, the dandelion/elephant lens is highly contextual. A whole company or an organization or even a product is not beholden to just one strategy. There can be multiple, complementary sets of strategies for the same product. 

    If I am building a REPL environment, I am clearly exercising the dandelion strategy in relation to its customers. I want ideas my customers have to be easily copyable, discoverable, fast to first results, etc. See the Interest, Legibility, Velocity, Access conditions I outlined earlier.

    However, when considering how to organize the development of this REPL environment itself (all the infrastructure and tooling that goes into creating a dandelion field for others), I am likely to take an elephant strategy. I would want capabilities that enable me to build upon my idea, not continue to reinvent it from scratch every few months. I will seek higher reliability, more features, rigorous processes, and increasingly more powerful capabilities – the outcome of the Stability, Breadth, Rigor, and Power conditions.

    Just like with any strategy, these are subject to becoming embodied. This is why I keep harping on about conditions. Us choosing to employ a given strategy is not a simple decision. It is a matter of the environment in which this decision is made. It is our environment that enables us to choose a strategy – or prevents us from doing so.

    Here’s one way to think of it. Our strategy is an aggregate of the moves we individually make. If most of us are making dandelion moves (rapidly mutating ideas we discover, generating new ones without holding on to the old ones), we are in the dandelion environment. If instead, we seem to be making mostly elephant moves (collectively reinforcing one big idea, making it richer, more nuanced, more thorough, etc.), we are in the elephant environment. 

    In either case, no matter how hard our leaders may call on us to change a strategy from the one we’ve currently embraced, we will only be able to produce gnarly beasts: dandelions with elephant trunks, or elephants made of pappus.

    The contextual quality of the lens allows us to use it to spot inconsistencies of our intentions with our conditions. Once spotted, these inconsistencies can offer a lot of insight on what nudges to make to the cone of embodied strategy.

    This framing of strategy challenges feels more hopeful to me. Instead of looking for someone to blame, look for the conditions that are present and whether or not these conditions are mismatched with the intention. If there is a distinct mismatch, look for ways to change conditions to align better with the desired outcomes.

  • Convergent innovation

    As a sort of thought appetizer, here’s a vignette attempting to intersect the divergent/convergent thinking frame and … you guessed it, dandelions and elephants.

    While exploring idea generation strategies, I realized that I’ve confused the opposite of divergent thinking with the lack of new ideas. That does not seem to be the case. There is a really simple 2×2 that helps illustrate that. Horizontally, we have dandelions and elephants, and vertically – the rate of new ideas that we want to encourage. Some organizations, for example, don’t want any new ideas altogether (low rate), while others have an existential need for new ideas (high rate).

    The upper quadrants make sense: we have our dandelion-fueled exploration and improvements to existing ideas through the elephant strategy. The bottom-right quadrant shows up as orthodoxy – when new ideas are unwanted because we believe that there’s already one idea that is understood as well as it could be. The final, bottom-left quadrant is that of the idea desert: the absence of ideas altogether. This quadrant rarely has permanent occupants, because we humans tend to have ideas – however, we might occasionally visit it when we’re lost and disoriented.

    If I were to explore how an organization might move from quadrant to quadrant, there appear to be four clockwise motions, from one quadrant to another. When our process of ⛵exploration (the top-left quadrant) yields many interesting ideas, organizations tend to shift into the convergent posture: start looking for one idea that will become bigger than others. A typical call here is “more wood behind fewer arrows” and conversations about prioritization and hard decisions. Convergence leads to the top-right quadrant of 📈 improvements, where most of our efforts are invested into reinforcing the idea.

    Thinking in this quadrant does not need to be boring and uncreative. For example, one of my colleagues, upon joining our team, spent a bunch of time looking through the code and spotted a bug: in handling mouse events, we would sometimes hit an O(n2) condition – and that was unnecessary. Their first commit to the codebase was a performance breakthrough. When I think of innovation in the top-right quadrant, I keep recalling that bit of code. It was clearly a novel insight, yet it also demonstrably improved the state of the bigger idea.

    Unfortunately, our residence in this quadrant usually comes to an end due to the consolidation move. In an effort to better protect the value contained within the big idea, organizations usually shift to the quadrant of ☝️orthodoxy, where any new ideas are viewed as mostly distractions. There’s so much stuff to do already! The problem lists are long and the issues are well-known. Let’s just keep on fixing them, shall we?

    As you may suspect from the flow of this story, the discouraging of new ideas eventually triggers the next move: obsolescence. This move puts us into the 🏜 idea desert. We know that the old big idea no longer works, but we have nothing else to hang onto.  The discomfort of this quadrant acts a powerful motivator, manifesting as the divergence move that propels us back into the  ⛵exploration quadrant. This move might coincide with the demise of the organization, when the divergence acts as a pulling-apart force of ideas, each stakeholder pursuing their own.

    If we are to believe this tall tale, we can see that divergence and convergence are somewhat orthogonal moves. The divergence is mostly about moving from zero to many ideas while relying on dandelion strategies. 

    On the other hand, the convergence is a qualitative shift. It does not change how much idea-generation we do, but rather whether or not these ideas are independent from each other. A productive convergence is the one where people ideate while building on each other’s ideas, improving upon one unifying big idea – the elephant strategy.

    This might not be news to you at all, but this was a pretty useful insight for me. When planning the next diverge/converge exercise:

    • To get divergence truly going, put people in the idea desert. Let them have the sense that all of their previous, strongly-held beliefs might not be as sound and safe as they seem. This might mean not letting people prepare for the exercise or even creating an idea parking lot that is filled at the beginning with the ideas that we already have.
    • Recognize the creative part of convergent thinking. Let people continue ideate, but shift the constraints of the exercise to encourage the clumping of ideas together. Avoid prematurely collapsing the process straight toward the orthodoxy. 

    Convergence doesn’t have to be a boring prioritize-and-cut procedure. It can be fun – and who knows, maybe produce new ideas that didn’t pop up during the divergence part of the exercise?

  • Generating ideas and strategy coherence

    I’ve been talking about dandelions and elephants for a while now, and yes, it may seem like I’ve gone a bit nuts. Oh well. It’s just that it’s such a good framing and I keep finding uses for it nearly every day. When applied to ideas, r/K-selection strategies seem to be uncommonly generative.

    It all begins with a question: what kind of new ideas do we want to produce? Do we want a collection of different, independent ideas or do we want each idea to improve upon some larger idea?

    What I like about these questions is that they are objective-agnostic. They don’t ask “what do you want to achieve?” or “where do you want to go?” Instead, they require us to choose the means to generate ideas. And strategy is all about the means. In the field where I work, strategy is also about generating new ideas.

    Here’s the thing. In software engineering (as likely in many technology fields), more often than not, we don’t know what the path to our objective will look like. Heck, most of the time we don’t even have a clear sense of what the objective will look like. This is assuredly not a “let’s plan all steps in advance” process. The fog of uncertainty is right there in front of us. 

    If we are to navigate toward it, we must be prepared to shift course, to adjust, to learn on the spot about the next step, make it, learn again, and so on. And to do this well, we need new ideas. Our strategy must count on us continuously producing these new ideas – and applying them. In this way, my ramblings about dandelions and elephants aren’t fun side metaphors. They are the essence of business.

    Summoning my inner Rumelt and putting things perhaps overly bluntly, an organization can only be effective at setting a strategy and actually following through when it is intentional about creating conditions for generating ideas. While it’s not the only crucial ingredient, the organization that doesn’t have it will suffer from strategic incoherence.

    A team may accept as a truism that bottom-up cultures are superior to top-down cultures. And yes, if we are setting out to explore a large space of unknown untapped potential, then we probably want to create conditions for a dandelion strategy. The bottom-up culture has them: individual incentives (Interest), small teams, short-term objectives (Legibility), independent decision-making (Velocity) and non-hierarchical structure and mobility (Access).

    However, when we’re endeavoring to care for one big idea, we likely want the conditions to encourage the elephant strategy: more structured and predictable organization and incentives (Stability), care and accountability in decision-making (Breadth), comprehensive processes and long-term thinking (Rigor), and concentrated points of organizational control (Power). These are a depiction of the top-down culture.

    If we set out to do something that calls for an elephant strategy, yet the culture we have is a bottom-up one, we will have strategically incoherent outcomes (I called them the “pappus elephants” in the previous post). Our bottom-up culture will suddenly snag us like a trap, with coordination headwinds becoming universally felt and recognized. Things that worked really well for us before, like emphasizing individual impact in our incentive structures, will become a source of pain: why are our teammates acting in such a self-interested way?! Well… maybe because that was a good thing when we needed a dandelion strategy?

    Even when the need to pursue a multi-year objective becomes existential, the dandelion conditions will keep blowing us off course: multi-year ideas will be simply swept away by the churn of the quarterly objective-setting and obsessive focus on individual impact. In a dandelion culture, when given a chance to make a dandelion move, most folks will take it. When strategy is incoherent, one can be a superstar while directly contributing to the team’s demise. 

    Perhaps even more bizarrely, by all accounts of witnesses, these efforts will look like elephants – until they disappear in a puff. It is in everyone’s interest to create a perception that they are indeed operating in an elephant factory, despite all the dandelion moves they are making. 

    When caught in this condition inconsistency, the long-term projects within this organization will inevitably find themselves in a weird cycle: set out to do big things, fail to articulate them clearly, struggle to do something very ambitious, get distracted, then quietly discontinue the effort, unable to examine what happened due to the deep sense of shame that follows – only to try again soon thereafter. When underlying conditions allow only dandelion-like moves, trying to choose an elephant strategy is a tough proposition.

    The variables and symptoms might vary, but the equation will remain the same. If they sound at all familiar, consider asking different questions to get to a more productive conversation about incentives, culture, structure, and practices. What are our current conditions for generating new ideas? Do they lean dandelion or elephant? How might they be inconsistent with our desired outcomes?

  • Dandelion/Elephant Strategy Rubric

    I discussed the nature of conditions that encourage dandelion or elephant strategies, but I only skimmed a little bit the topic of why one would choose these strategies. Here’s a sketch of digging a bit more in depth of this particular “why”.

    A rubric that we want to use here stems from the observation of environments leading to the thriving of the organizations that chose either strategy. The environment must be complementary to the strategy.

    Elephants and other K-selected species choose their strategy because the environment makes this strategy viable: there is a well-defined, stable niche into which the overall shape of the species’ activities fits. This niche is usually crowded: there are other species within that niche, and the game of life is mostly about optimizing the shape to vie for bettering other species. In this way, the environment of an elephant strategy is shape-focused.

    This helps us define a rule of thumb for choosing to pursue the elephant strategy: if the shape of our technology or product is well-known and well-understood (and thus strongly expected) by our market, the elephant strategy is likely warranted.

    The word “shape” here is meant in a broader sense: it’s not necessarily the physical dimensions of some object. Rather, it’s the whole set of expectations and constraints that define the niche in which we’re playing.

    Using this definition, we can see that the basic shape of a mobile phone is becoming more and more entrenched in the mindshare of the world’s population. It’s the iconic iPhone shape that Steve Jobs revealed in 2007. There’s an expectation of a touch screen and a camera, of GPS, and various other sensors that are considered table stakes by the users. Should a company decide to ship a phone that does not fit into that shape, it will be playing at a different, less populated table.

    Everyone who wants to live in this niche must be engaged in a strongly elephant-leaning strategy: year over year, continuous, incremental improvement of what can fit into the shape defined by the niche. We think of elephants as being slow, and that’s where the metaphor falls a bit short. Pursuing the elephant strategy is all about steadily accumulating mass and gaining velocity through momentum. Maybe thinking about how a capital ship reaches their tremendous speed would help here. It’s the game of consistency and compounding momentum, not leaps of faith or moonshots.

    Prioritizing while pursuing elephant strategy, we must ask ourselves these questions: does this work help us accumulate momentum over the long term? Does it contribute to optimizing for the shape in the niche we’re playing? Does it help us better understand the nuanced nooks and crannies of the shape that might be critical for the relentless striving of shape-fitting? 

    Having impact in pursuit of elephant strategy means contributing to gaining momentum. If I did something impressive, but not aligned with this larger goal, I probably wasted my energy – or worse yet, slowed down the ship. As I described before, top-down cultures tend to be effective here: it is easier to understand what to align with.

    As a thought experiment, compare two teams: in one team, planning happens as a top-down process, and in another – bottom-up. In the top-down team, the direction is clearly stated at the beginning of the planning, and all team members must shape their work to align with this direction. Conversely, in a bottom-up planning process, the members supply what they are planning to do and then the sum vector of this work determines where the team will go. Which team will have a greater consistency of momentum accumulation over time?

    On the other hand, dandelions and the r-selected species they represent choose their strategy because the space in front of them is wide open. This typically happens when a technological breakthrough suddenly provides the means to build something new, but the actual product-market fit – the shape of the product or technology – is entirely unclear. There are no right or wrong answers – yet. There aren’t clearly defined constraints and the perceived limits seem to give when pushed on. Everyone else is doing roughly the same thing: stumbling around to find out what the heck this new space is all about.

    To turn it into a rule of thumb for choosing the dandelion strategy: if the technology or product space is ill-defined and little-understood, pick the dandelion strategy.

    I already used the AI-generated media space as the currently-unfolding example of this scenario, and it’s still very relevant. It is not knowable where this thing will go. Early attempts to discern constraints or define a shape will likely look foolish in the long run.

    It is fairly easy to see how trying to build a new dreadnought or turn the existing ship toward this momentum are both activities fraught with peril. When we don’t know where dandelion seeds will land, we are better off letting go of our habit of predicting the outcomes.

    Instead of investing into enterprise-strength, scalable software, we can be better off with duct tape and popsicle sticks when adopting the dandelion strategy. Throw something together, make it go, ship it, and see what happens. Do it again. And again. Watch carefully for surprising developments. If our tiny app suddenly gains a small following – that’s something. Avoid presuming that we know what this “something” is. Very often, in a new space, even when given an opportunity to tell us directly, our users might not be able to articulate it, either. Revealed preferences rule over the stated ones. The trick is to keep trying and learning and making sure that the learning propagates to some common pool of wisdom.

    In such an environment, bottom-up cultures work amazingly well. Returning to the earlier thought experiment, can you see how the fortunes of these teams will be reversed? The top-down team will form a tight fist, punch hard in one direction … to never be heard from again. In contrast, the bottom-up team will diverge in a cloud of divergent direction vectors, thus maximizing their chances of stumbling onto a fertile niche.

    Here, being impactful means uncovering something interesting and surprising as quickly as possible – and bringing it back to the team. Trying and failing is just as useful, because it uncovers where not to go, or go about differently. The key difference between measuring the impact for the elephant strategy is in contributing to the common pool of knowledge while exploring a direction that’s different from the rest of the team. This can feel rather unintuitive: members reinforcing each other’s approaches can be a source of blindness when applying the dandelion strategy. The way to structure incentives here is to emphasize individual agency while rewarding contributions to collective knowledge of the space.

    How does a bottom-up team prioritize? In an uncertain environment, prioritization is emergent. There aren’t well-defined metrics and clear lines to cut. Instead, the team’s stumbling into novelty is the source of knowing what’s important, leading to recurring waves of swarming and scattering. This may feel rather mercurial and drive some engineers and program managers nuts. The trick here is to zero in not on the ever-moving objects of prioritization, but rather on whether the information about these objects is flowing as quickly and clearly as possible.

    To summarize these two rules of thumb, I will bring them together into a rubric. Ask these two questions:

    1. Is the shape of the product/technology niche in which we are playing well-understood? 
    2. Are we playing in (or closely adjacent to) the space that just opened up because of a recent technological breakthrough?

    The answers form a 2×2.

    For the environments where the shape is well-known, with no new space opening up, we are looking at a strongly elephant-leaning strategy. Get that colossal ship going and keep it rolling. Don’t get discouraged when first outcomes are unsatisfying. Elephant calves need nursing and care.

    If the answer to both questions is “yes”, we’re probably seeing increasing potential for a new product category in a previously-stable space. Something curious will happen soon, and we don’t want to miss it. Deploy the “fuzzy elephant” stance: structure most of the team to adopt the elephant-leaning strategy, with a modest-sized group making dandelion moves. Given the recent rate of technological advance, this is an effective posture for any large player: there will always be an opportunity of surprise.

    The full-on dandelion approach is warranted in the presence of a technological breakthrough in a brand new area with few well-defined niches. Manage divergence and get that insight flow going – who will be the first player to spot a niche?

    The final quadrant is a bit puzzling for me. If the shape is not known and there aren’t any breakthroughs, why are we playing here? It feels like there might not be enough evolutionary pressure to get the selection process going – which means that if we find ourselves in this space, we are better off looking for a way out.

%d bloggers like this: